Post pobrano z: A Primer on the Different Types of Browser Storage

In back-end development, storage is a common part of the job. Application data is stored in databases, files in object storage, transient data in caches… there are seemingly endless possibilities for storing any sort of data. But data storage isn’t limited only to the back end. The front end (the browser) is equipped with many options to store data as well. We can boost our application performance, save user preferences, keep the application state across multiple sessions, or even different computers, by utilizing this storage.

In this article, we will go through the different possibilities to store data in the browser. We will cover three use cases for each method to grasp the pros and cons. In the end, you will be able to decide what storage is the best fit for your use case. So let’s start!

The localStorage API

localStorage is one of the most popular storage options in the browser and the go-to for many developers. The data is stored across sessions, never shared with the server, and is available for all pages under the same protocol and domain. Storage is limited to ~5MB.

Surprisingly, the Google Chrome team doesn’t recommend using this option as it blocks the main thread and is not accessible to web workers and service workers. They launched an experiment, KV Storage, as a better version, but it was just a trial that doesn’t seem to have gone anywhere just yet.

The localStorage API is available as window.localStorage and can save only UTF-16 strings. We must make sure to convert data to strings before saving it into localStorage. The main three functions are:

• setItem('key', 'value')
• getItem('key')
• removeItem('key')

They’re all synchronous, which makes it simple to work with, but they block the main thread.

It’s worth mentioning that localStorage has a twin called sessionStorage. The only difference is that data stored in sessionStorage will last only for the current session, but the API is the same.

Let’s see it in action. The first example demonstrates how to use localStorage for storing the user’s preferences. In our case, it’s a boolean property that turns on or off the dark theme of our site.

You can check the checkbox and refresh the page to see that the state is saved across sessions. Take a look at the save and load functions to see how I convert the value to string and how I parse it. It’s important to remember that we can store only strings.

This second example loads Pokémon names from the PokéAPI.

We send a GET request using fetch and list all the names in a ul element. Upon getting the response, we cache it in the localStorage so our next visit can be much faster or even work offline. We have to use JSON.stringify to convert the data to string and JSON.parse to read it from the cache.

In this last example, I demonstrate a use case where the user can browse through different Pokémon pages, and the current page is saved for the next visits.

The issue with localStorage, in this case, is that the state is saved locally. This behavior doesn’t allow us to share the desired page with our friends. Later, we will see how to overcome this issue.

We will use these three examples in the next storage options as well. I forked the Pens and just changed the relevant functions. The overall skeleton is the same for all methods.

The IndexedDB API

IndexedDB is a modern storage solution in the browser. It can store a significant amount of structured data — even files, and blobs. Like every database, IndexedDB indexes the data for running queries efficiently. It’s more complex to use IndexedDB. We have to create a database, tables, and use transactions.

Compared to localStorage , IndexedDB requires a lot more code. In the examples, I use the native API with a Promise wrapper, but I highly recommend using third-party libraries to help you out. My recommendation is localForage because it uses the same localStorage API but implements it in a progressive enhancement manner, meaning if your browser supports IndexedDB, it will use it; and if not, it will fall back to localStorage.

Let’s code, and head over to our user preferences example!

idb is the Promise wrapper that we use instead of working with a low-level events-based API. They’re almost identical, so don’t worry. The first thing to notice is that every access to the database is async, meaning we don’t block the main thread. Compared to localStorage, this is a major advantage.

We need to open a connection to our database so it will be available throughout the app for reading and writing. We give our database a name, my-db, a schema version, 1, and an update function to apply changes between versions. This is very similar to database migrations. Our database schema is simple: only one object store, preferences. An object store is the equivalent of an SQL table. To write or read from the database, we must use transactions. This is the tedious part of using IndexedDB. Have a look at the new save and load functions in the demo.

No doubt that IndexedDB has much more overhead and the learning curve is steeper compared to localStorage. For the key value cases, it might make more sense to use localStorage or a third-party library that will help us be more productive.

Application data, such as in our Pokémon example, is the forte of IndexedDB. You can store hundreds of megabytes and even more in this database. You can store all the Pokémon in IndexedDB and have them available offline and even indexed! This is definitely the one to choose for storing app data.

I skipped the implementation of the third example, as IndexedDB doesn’t introduce any difference in this case compared to localStorage. Even with IndexedDB, the user will still not share the selected page with others or bookmark it for future use. They’re both not the right fit for this use case.

Cookies

Using cookies is a unique storage option. It’s the only storage that is also shared with the server. Cookies are sent as part of every request. It can be when the user browses through pages in our app or when the user sends Ajax requests. This allows us to create a shared state between the client and the server, and also share state between multiple applications in different subdomains. This is not possible by other storage options that are described in this article. One caveat: cookies are sent with every request, which means that we have to keep our cookies small to maintain a decent request size.

The most common use for cookies is authentication, which is out of the scope of this article. Just like the localStorage, cookies can store only strings. The cookies are concatenated into one semicolon-separated string, and they are sent in the cookie header of the request. You can set many attributes for every cookie, such as expiration, allowed domains, allowed pages, and many more.

In the examples, I show how to manipulate the cookies through the client-side, but it’s also possible to change them in your server-side application.

Saving the user’s preferences in a cookie can be a good fit if the server can utilize it somehow. For example, in the theme use case, the server can deliver the relevant CSS file and reduce potential bundle size (in case we’re doing server-side-rendering). Another use case might be to share these preferences across multiple subdomain apps without a database.

Reading and writing cookies with JavaScript is not as straightforward as you might think. To save a new cookie, you need to set document.cookie — check out the save function in the example above. I set the dark_theme cookie and add it a max-age attribute to make sure it will not expire when the tab is closed. Also, I add the SameSite and Secure attributes. These are necessary because CodePen uses iframe to run the examples, but you will not need them in most cases. Reading a cookie requires parsing the cookie string.

A cookie string looks like this:

key1=value1;key2=value2;key3=value3

So, first, we have to split the string by semicolon. Now, we have an array of cookies in the form of key1=value1, so we need to find the right element in the array. In the end, we split by the equal sign and get the last element in the new array. A bit tedious, but once you implement the getCookie function (or copy it from my example :P) you can forget it.

Saving application data in a cookie can be a bad idea! It will drastically increase the request size and will reduce application performance. Also, the server cannot benefit from this information as it’s a stale version of the information it already has in its database. If you use cookies, make sure to keep them small.

The pagination example is also not a good fit for cookies, just like localStorage and IndexedDB. The current page is a temporary state that we would like to share with others, and any of these methods do not achieve it.

URL storage

URL is not a storage, per se, but it’s a great way to create a shareable state. In practice, it means adding query parameters to the current URL that can be used to recreate the current state. The best example would be search queries and filters. If we search the term flexbox on CSS-Tricks, the URL will be updated to https://css-tricks.com/?s=flexbox. See how easy it is to share a search query once we use the URL? Another advantage is that you can simply hit the refresh button to get newer results of your query or even bookmark it.

We can save only strings in the URL, and its maximum length is limited, so we don’t have so much space. We will have to keep our state small. No one likes long and intimidating URLs.

Again, CodePen uses iframe to run the examples, so you cannot see the URL actually changing. Worry not, because all the bits and pieces are there so you can use it wherever you want.

We can access the query string through window.location.search and, lucky us, it can be parsed using the URLSearchParams class. No need to apply any complex string parsing anymore. When we want to read the current value, we can use the get function. When we want to write, we can use set. It’s not enough to only set the value; we also need to update the URL. This can be done using history.pushState or history.replaceState, depending on the behavior we want to accomplish.

I wouldn’t recommend saving a user’s preferences in the URL as we will have to add this state to every URL the user visits, and we cannot guarantee it; for example, if the user clicks on a link from Google Search.

Just like cookies, we cannot save application data in the URL as we have minimal space. And even if we did manage to store it, the URL will be long and not inviting to click. Might look like a phishing attack of sorts.

Just like our pagination example, the temporary application state is the best fit for the URL query string. Again, you cannot see the URL changes, but the URL updates with the ?page=x query parameter every time you click on a page. When the web page loads, it looks for this query parameter and fetches the right page accordingly. Now we can share this URL with our friends so they can enjoy our favorite Pokémon.

Cache API

Cache API is a storage for the network level. It is used to cache network requests and their responses. The Cache API fits perfectly with service workers. A service worker can intercept every network request, and using the Cache API, it can easily cache both the requests. The service worker can also return an existing cache item as a network response instead of fetching it from the server. By doing so, you can reduce network load times and make your application work even when offline. Originally, it was created for service workers but in modern browsers the Cache API is available also in window, iframe, and worker contexts as-well. It’s a very powerful API that can improve drastically the application user experience.

Just like IndexedDB the Cache API storage is not limited and you can store hundreds of megabytes and even more if you need to. The API is asynchronous so it will not block your main thread. And it’s accessible through the global property caches.

To read more about the Cache API, the Google Chrome team has made a great tutorial.

Chris created an awesome Pen with a practical example of combining service workers and the Cache API.

Bonus: Browser extension

If you build a browser extension, you have another option to store your data. I discovered it while working on my extension, daily.dev. It’s available via chrome.storage or browser.storage, if you use Mozilla’s polyfill. Make sure to request a storage permission in your manifest to get access.

There are two types of storage options, local and sync. The local storage is self-explanatory; it means it isn’t shared and kept locally. The sync storage is synced as part of the Google account and anywhere you install the extension with the same account this storage will be synced. Pretty cool feature if you ask me. Both have the same API so it’s super easy to switch back-and-forth, if needed. It’s async storage so it doesn’t block the main thread like localStorage. Unfortunately, I cannot create a demo for this storage option as it requires a browser extension but it’s pretty simple to use, and almost like localStorage. For more information about the exact implementation, refer to Chrome docs.

Conclusion

The browser has many options we can utilize to store our data. Following the Chrome team’s advice, our go-to storage should be IndexedDB. It’s async storage with enough space to store anything we want. localStorage is not encouraged, but is easier to use than IndexedDB. Cookies are a great way to share the client state with the server but are mostly used for authentication.

If you want to create pages with a shareable state such as a search page, use the URL’s query string to store this information. Lastly, if you build an extension, make sure to read about chrome.storage.

---

The post A Primer on the Different Types of Browser Storage appeared first on CSS-Tricks.

You can support CSS-Tricks by being an MVP Supporter.

Post pobrano z: xm

This is a neat little HTML preprocessor from Giuseppe Gurgone. It has very few features, but one of them is HTML includes, which is something I continue to be baffled that HTML doesn’t support natively. There are loads of ways to handle it. I think it’s silly that it’s been consistently needed for decades and HTML could evolve to support it but hasn’t. So anyway, enter another option for handling it.

📢 Today I am open sourcing ✨ ₪ xm ✨ a tiny compiler for HTML that adds support for

imports
slots & fills
a portal to markdown

🔗 https://t.co/mhrlVGg3Wp
⌨️ npm i -g xm && xm dev pic.twitter.com/pYN9RpodsW

— Giuseppe (@giuseppegurgone) September 15, 2020

What is extra neat is that it’s not just includes, but templating with includes in a really clean way. If this was Nunjucks, they solve that by creating a template.njk like…

{% block header %}
  This is the default (overridable) header.
{% endblock %}
<footer>
  {% block footer %}
    This is the default (overridable) footer.
  {% endblock %}
</footer>

And then your actual pages use that template like…

{% extends "parent.html" %}
{% block footer %}
  Special footer for this page.
{% endblock %}

In xm, the syntax stays HTML-y, which is nice. So this template.html…

<slot name="header"></slot>
<footer>
  <slot name="footer"></slot>
</footer>

…gets used on a page like this:

<import src="template.html">
  <fill name="header">Custom Header</fill>
  <fill name="footer">
    <p>Custom footer</p>
  </fill>
</import>

Very clean. The additional fact that you can arbitrarily chuck a <markdown> tag anywhere you want and use Markdown within it is extra handy.

---

The post xm appeared first on CSS-Tricks.

You can support CSS-Tricks by being an MVP Supporter.

Post pobrano z: How to Think Like a Front-End Developer

This is an extended version of my essay “When front-end means full-stack” which was published in the wonderful Increment magazine put out by Stripe. It’s also something of an evolution of a couple other of my essays, “The Great Divide” and “Ooops, I guess we’re full-stack developers now.”

Looking back at the courses I chose from middle school through college, I bounced back and forth between computer-focused classes and art-focused classes, so I suppose it’s no surprise I found a way to do both as a career.

The term “Front-End Developer” is fairly well-defined and understood. For one, it’s a job title. I’ll bet some of you literally have business cards that say it on there, or some variation like: “Front-End Designer,” “UX Developer,” or “UI Engineer.” The debate around what those mean isn’t particularly interesting to me. I find that the roles are so varied from job-to-job and company-to-company that job titles will never be enough to describe things. Getting this job is more about demonstrating you know what you’re doing more than anything else¹.

The title variations are just nuance. The bigger picture is that as long as the job is building websites, front-enders are focused on the browser. Quite literally:

• front-end = browsers
• back-end = servers

Even as the job has changed over the decades, that distinction still largely holds.

As “browser people,” there are certain truths that come along for the ride. One is that there is a whole landscape of different browsers and, despite the best efforts of standards bodies, they still behave somewhat differently. Just today, as I write, I dealt with a bug where a date string I had from an API was in a format such that Firefox threw an error when I tried to use the .toISOString() JavaScript API on it, but was fine in Chrome. That’s just life as a front-end developer. That’s the job.

The most important aspect of the job? The people that use these browsers. That’s why we’re building things at all. These are the people I’m trying to impress with my mad CSS skills. These are the people I’m trying to get to buy my widget. Who all my business charts hinge upon. Who’s reaction can sway my emotions like yarn in the breeze. These users, who we put on a pedestal for good reason, have a much wider landscape than the browsers do. They speak different languages. They want different things. They are trying to solve different problems. They have different physical abilities. They have different levels of urgency. Again, helping them is firmly in the hands of front-end developers. There is very little in between the characters we type into our text editors and the users for whom we wish to serve.

Being a front-end developer puts us on the front lines between the thing we’re building and the people we’re building it for, and that’s a place some of us really enjoy being.

That’s some weighty stuff, isn’t it? I haven’t even mentioned React yet.

The “we care about the users” thing might feel a little precious. I’d think in a high functioning company, everyone would care about the users, from the CEO on down. It’s different, though. When we code a <button>, we’re quite literally putting a button into a browser window that users directly interact with. When we adjust a color, we’re adjusting exactly what our sighted users see when they see our work.

---

We established that front-end developers are browser people. The job is making things work well in browsers. So we need to understand the languages browsers speak, namely: HTML, CSS, and JavaScript². And that’s not just me being some old school fundamentalist; it’s through a few decades of everyday front-end development work that knowing those base languages is vital to us doing a good job. Even when we don’t work directly with them (HTML might come from a template in another language, CSS might be produced from a preprocessor, JavaScript might be mostly written in the parlance of a framework), what goes the browser is ultimately HTML, CSS, and JavaScript, so that’s where debugging largely takes place and the ability of the browser is put to work.

CSS will always be my favorite and HTML feels like it needs the most love — but JavaScript is the one we really need to examine The last decade has seen JavaScript blossom from a language used for a handful of interactive effects to the predominant language used across the entire stack of web design and development. It’s possible to work on websites and writing nothing but JavaScript. A real sea change.

JavaScript is all-powerful in the browser. In a sense, it supersedes HTML and CSS, as there is nothing either of those languages can do that JavaScript cannot. HTML is parsed by the browser and turned into the DOM, which JavaScript can also entirely create and manipulate. CSS has its own model, the CSSOM, that applies styles to elements in the DOM, which JavaScript can also create and manipulate.

This isn’t quite fair though. HTML is the very first file that browsers parse before they do the rest of the work needed to build the site. That firstness is unique to HTML and a vital part of making websites fast.

In fact, if the HTML was the only file to come across the network, that should be enough to deliver the basic information and functionality of a site.

That philosophy is called Progressive Enhancement. I’m a fan, myself, but I don’t always adhere to it perfectly. For example, a <form> can be entirely functional in HTML, when it’s action attribute points to a URL where the form can be processed. Progressive Enhancement would have us build it that way. Then, when JavaScript executes, it takes over the submission and has the form submit via Ajax instead, which might be a nicer experience as the page won’t have to refresh. I like that. Taken further, any <button> outside a form is entirely useless without JavaScript, so in the spirit of Progressive Enhancement, I should wait until JavaScript executes to even put that button on the page at all (or at least reveal it). That’s the kind of thing where even those of us with the best intentions might not always toe the line perfectly. Just put the button in, Sam. Nobody is gonna die.

JavaScript’s all-powerfulness makes it an appealing target for those of us doing work on the web — particularly as JavaScript as a language has evolved to become even more powerful and ergonomic, and the frameworks that are built in JavaScript become even more-so. Back in 2015, it was already so clear that JavaScript was experiencing incredible growth in usage, Matt Mullenweg, co-founder of WordPress, gave the developer world homework: “Learn JavaScript Deeply”³. He couldn’t have been more right. Half a decade later, JavaScript has done a good job of taking over front-end development. Particularly if you look at front-end development jobs.

While the web almanac might show us that only 5% of the top-zillion sites use React compared to 85% including jQuery, those numbers are nearly flipped when looking around at front-end development job requirements.

I’m sure there are fancy economic reasons for all that, but jobs are as important and personal as it gets for people, so it very much matters.

---

So we’re browser people in a sea of JavaScript building things for people. If we take a look at the job at a practical day-to-day tasks level, it’s a bit like this:

Just that first bullet point feels like a college degree to me. Taken together, all of those points certainly do.

This whole list is a bit abstract though, so let’s apply it to something we can look at. What if this website was our current project?

Our brains and fingers go wild!

I could go on and on. That’s how front-end developers think, at least in my experience and in talking with my peers.

A lot of those things have been our jobs forever though. We’ve been asking and answering these questions on every website we’ve built for as long as we’ve been doing it. There are different challenges on each site, which is great and keeps this job fun, but there is a lot of repetition too.

Allow me to get around to the title of this article. 

I like it, this idea of components. It allows you and your team to build the abstractions that make the most sense to you and what you are building.

Your Card component does all the stuff your card needs to do. Your Form component does forms how your website needs to do forms. But it’s a new concept to old developers like me. Components in JavaScript have taken hold in a way that components on the server-side never did. I’ve worked on many a WordPress website where the best I did was break templates into somewhat arbitrary include() statements. I’ve worked on Ruby on Rails sites with partials that take a handful of local variables. Those are useful for building re-usable parts, but they are a far cry from the robust component models that JavaScript frameworks offer us today.

All this custom component creation makes me a site-level architect in a way that I didn’t use to be. Here’s an example. Of course I have a Button component. Of course I have an Icon component. I’ll use them in my Card component. My Card component lives in a Grid component that lays them out and paginates them. The whole page is actually built from components. The Header component has a SearchBar component and a UserMenu component. The Sidebar component has a Navigation component and an Ad component. The whole page is just a special combination of components, which is probably based on the URL, assuming I’m all-in on building our front-end with JavaScript. So now I’m dealing with URLs myself, and I’m essentially the architect of the entire site. [Sweats profusely]

Like I told ya, a whole pile of new responsibility.

Components that are in charge of displaying content are almost certainly not hard-coded with data in them. They are built to be templates. They are built to accept data and construct themselves based on that data. In the olden days, when we were doing this kind of templating, the data has probably already arrived on the page we’re working on. In a JavaScript-powered app, it’s more likely that that data is fetched by JavaScript. Perhaps I’ll fetch it when the component renders. In a stack I’m working with right now, the front end is in React, the API is in GraphQL and we use Apollo Client to work with data. We use a special “hook” in the React components to run the queries to fetch the data we need, and another special hook when we need to change that data. Guess who does that work? Is it some other kind of developer that specializes in this data layer work? No, it’s become the domain of the front-end developer.

Speaking of data, there is all this other data that a website often has to deal with that doesn’t come from a database or API. It’s data that is really only relevant to the website at this moment in time.

Front-end developers have been dealing with that kind of state for a long time, but it’s exactly this kind of state that has gotten us into trouble before. A modal dialog can be open with a simple modifier class like <div class="modal is-open"> and toggling that class is easy enough with .classList.toggle(".is-open"); But that’s a purely visual treatment. How does anything else on the page know if that modal is open or not? Does it ask the DOM? In a lot of jQuery-style apps of yore, yes, it would. In a sense, the DOM became the “source of truth” for our websites. There were all sorts of problems that stemmed from this architecture, ranging from a simple naming change destroying functionality in weirdly insidious ways, to hard-to-reason-about application logic making bug fixing a difficult proposition.

Front-end developers collectively thought: what if we dealt with state in a more considered way? State management, as a concept, became a thing. JavaScript frameworks themselves built the concept right in, and third-party libraries have paved and continue to pave the way. This is another example of expanding responsibility. Who architects state management? Who enforces it and implements it? It’s not some other role, it’s front-end developers.

There is expanding responsibility in the checklist of things to do, but there is also work to be done in piecing it all together. How much of this state can be handled at the individual component level and how much needs to be higher level? How much of this data can be gotten at the individual component level and how much should be percolated from above? Design itself comes into play. How much of the styling of this component should be scoped to itself, and how much should come from more global styles?

It’s no wonder that design systems have taken off in recent years. We’re building components anyway, so thinking of them systemically is a natural fit.

Let’s look at our design again:

A bunch of new thoughts can begin!

Those are all things that are in the domain of front-end developers these days, on top of everything that we already need to do. Executing the design, semantics, accessibility, performance… that’s all still there. You still need to be proficient in HTML, CSS, JavaScript, and how the browser works. Being a front-end developer requires a haystack of skills that grows and grows. It’s the natural outcome of the web getting bigger. More people use the web and internet access grows. The economy around the web grows. The capability of browsers grows. The expectations of what is possible on the web grows. There isn’t a lot shrinking going on around here.

We’ve already reached the point where most front-end developers don’t know the whole haystack of responsibilities. There are lots of developers still doing well for themselves being rather design-focused and excelling at creative and well-implemented HTML and CSS, even as job posts looking for that dwindle.

There are systems-focused developers and even entire agencies that specialize in helping other companies build and implement design systems. There are data-focused developers that feel most at home making the data flow throughout a website and getting hot and heavy with business logic. While all of those people might have “front-end developer” on their business card, their responsibilities and even expectations of their work might be quite different. It’s all good, we’ll find ways to talk about all this in time.

In fact, how we talk about building websites has changed a lot in the last decade. Some of my early introduction to web development was through WordPress. WordPress needs a web server to run, is written in PHP, and stores it’s data in a MySQL database. As much as WordPress has evolved, all that is still exactly the same. We talk about that “stack” with an acronym: LAMP, or Linux, Apache, MySQL and PHP. Note that literally everything in the entire stack consists of back-end technologies. As a front-end developer, nothing about LAMP is relevant to me.

But other stacks have come along since then. A popular stack was MEAN (Mongo, Express, Angular and Node). Notice how we’re starting to inch our way toward more front-end technologies? Angular is a JavaScript framework, so as this stack gained popularity, so too did talking about the front-end as an important part of the stack. Node and Express are both JavaScript as well, albeit the server-side variant.

The existence of Node is a huge part of this story. Node isn’t JavaScript-like, it’s quite literally JavaScript. It makes a front-end developer already skilled in JavaScript able to do server-side work without too much of a stretch.

“Serverless” is a much more modern tech buzzword, and what it’s largely talking about is running small bits of code on cloud servers. Most often, those small bits of code are in Node, and written by JavaScript developers. These days, a JavaScript-focused front-end developer might be writing their own serverless functions and essentially being their own back-end developer. They’ll think of themselves as full-stack developers, and they’ll be right.

Shawn Wang coined a term for a new stack this year: STAR or Design System, TypeScript, Apollo, and React. This is incredible to me, not just because I kind of like that stack, but because it’s a way of talking about the stack powering a website that is entirely front-end technologies. Quite a shift.

I apologize if I’ve made you feel a little anxious reading this. If you feel like you’re behind in understanding all this stuff, you aren’t alone.

In fact, I don’t think I’ve talked to a single developer who told me they felt entirely comfortable with the entire world of building websites. Everybody has weak spots or entire areas where they just don’t know the first dang thing. You not only can specialize, but specializing is a pretty good idea, and I think you will end up specializing to some degree whether you plan to or not. If you have the good fortune to plan, pick things that you like. You’ll do just fine.

The only constant in life is change.

– Heraclitus
    – Motivational Poster
        – Chris Coyier

---

---

The post How to Think Like a Front-End Developer appeared first on CSS-Tricks.

You can support CSS-Tricks by being an MVP Supporter.

Post pobrano z: Create an FAQ Slack app with Netlify functions and FaunaDB

Sometimes, when you’re looking for a quick answer, it’s really useful to have an FAQ system in place, rather than waiting for someone to respond to a question. Wouldn’t it be great if Slack could just answer these FAQs for us? In this tutorial, we’re going to be making just that: a slash command for Slack that will answer user FAQs. We’ll be storing our answers in FaunaDB, using FQL to search the database, and utilising a Netlify function to provide a serverless endpoint to connect Slack and FaunaDB.

Prerequisites

This tutorial assumes you have the following requirements:

• Github account, used to log in to Netlify and Fauna, as well as storing our code
• Slack workspace with permission to create and install new apps
• Node.js v12

Create npm package

To get started, create a new folder and initialise a npm package by using your package manager of choice and run npm init -y from inside the folder. After the package has been created, we have a few npm packages to install.

Run this to install all the packages we will need for this tutorial:

npm install express body-parser faunadb encoding serverless-http netlify-lambda

These packages are explained below, but if you are already familiar with them, feel free to skip ahead.

Encoding has been installed due to a plugin error occurring in @netlify/plugin-functions-core at the time of writing and may not be needed when you follow this tutorial.

Packages

Express is a web application framework that will allow us to simplify writing multiple endpoints for our function. Netlify functions require handlers for each endpoint, but express combined with serverless-http will allow us to write the endpoints all in one place.

Body-parser is an express middleware which will take care of the application/x-www-form-urlencoded data Slack will be sending to our function.

Faunadb is an npm module that allows us to interact with the database through the FaunaDB Javascript driver. It allows us to pass queries from our function to the database, in order to get the answers

Serverless-http is a module that wraps Express applications to the format expected by Netlify functions, meaning we won’t have to rewrite our code when we shift from local development to Netlify.

Netlify-lambda is a tool which will allow us to build and serve our functions locally, in the same way they will be built and deployed on Netlify. This means we can develop locally before pushing our code to Netlify, increasing the speed of our workflow.

Create a function

With our npm packages installed, it’s time to begin work on the function. We’ll be using serverless to wrap an express app, which will allow us to deploy it to Netlify later. To get started, create a file called netlify.toml, and add the following into it:

[build]
  functions = "functions"

We will use a .gitignore file, to prevent our node_modules and functions folders from being added to git later. Create a file called .gitignore, and add the following:

functions/

node_modules/

We will also need a folder called src, and a file inside it called server.js. Your final file structure should look like:

With this in place, create a basic express app by inserting the code below into server.js:

const express = require("express");
const bodyParser = require("body-parser");
const fauna = require("faunadb");
const serverless = require("serverless-http");
 
const app = express();
 
module.exports.handler = serverless(app);

Check out the final line; it looks a little different to a regular express app. Rather than listening on a port, we’re passing our app into serverless and using this as our handler, so that Netlify can invoke our function.

Let’s set up our body parser to use application/x-www-form-urlencoded data, as well as putting a router in place. Add the following to server.js after defining app: 

const router = express.Router();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use("/.netlify/functions/server", router);

Notice that the router is using /.netlify/functions/server as an endpoint. This is so that Netlify will be able to correctly deploy the function later in the tutorial. This means we will need to add this to any base URLs, in order to invoke the function.

Create a test route

With a basic app in place, let’s create a test route to check everything is working. Insert the following code to create a simple GET route, that returns a simple json object:

router.get("/test", (req, res) => {
 res.json({ hello: "world" });
});

With this route in place, let’s spin up our function on localhost, and check that we get a response. We’ll be using netlify-lambda to serve our app, so that we can imitate a Netlify function locally on port 9000. In our package.json, add the following lines into the scripts section:

"start": "./node_modules/.bin/netlify-lambda serve src",
   "build": "./node_modules/.bin/netlify-lambda build src"

With this in place, after saving the file, we can run npm start to begin netlify-lambda on port 9000.

The build command will be used when we deploy to Netlify later.

Once it is up and running, we can visit http://localhost:9000/.netlify/functions/server/test to check our function is working as expected.

The great thing about netlify-lambda is it will listen for changes to our code, and automatically recompile whenever we update something, so we can leave it running for the duration of this tutorial.

Start ngrok URL

Now we have a test route working on our local machine, let’s make it available online. To do this, we’ll be using ngrok, a npm package that provides a public URL for our function. If you don’t have ngrok installed already, first run npm install -g ngrok to globally install it on your machine. Then run ngrok http 9000 which will automatically direct traffic to our function running on port 9000.

After starting ngrok, you should see a forwarding URL in the terminal, which we can visit to confirm our server is available online. Copy this base URL to your browser, and follow it with /.netlify/functions/server/test. You should see the same result as when we made our calls on localhost, which means we can now use this URL as an endpoint for Slack!

Each time you restart ngrok, it creates a new URL, so if you need to stop it at any point, you will need to update your URL endpoint in Slack.

Setting up Slack

Now that we have a function in place, it’s time to move to Slack to create the app and slash command. We will have to deploy this app to our workspace, as well as making a few updates to our code to connect our function. For a more in depth set of instructions on how to create a new slash command, you can follow the official Slack documentation. For a streamlined set of instructions, follow along below:

Create a new Slack app

First off, let’s create our new Slack app for these FAQs. Visit https://api.slack.com/apps and select Create New App to begin. Give your app a name (I used Fauna FAQ), and select a development workspace for the app.

Create a slash command

After creating the app, we need to add a slash command to it, so that we can interact with the app. Select slash commands from the menu after the app has been created, then create a new command. Fill in the following form with the name of your command (I used /faq) as well as providing the URL from ngrok. Don’t forget to add /.netlify/functions/server/ to the end!

Install app to workspace

Once you have created your slash command, click on basic information in the sidebar on the left to return to the app’s main page. From here, select the dropdown “Install app to your workspace” and click the button to install it.

Once you have allowed access, the app will be installed, and you’ll be able to start using the slash command in your workspace.

Update the function

With our new app in place, we’ll need to create a new endpoint for Slack to send the requests to. For this, we’ll use the root endpoint for simplicity. The endpoint will need to be able to take a post request with application/x-www-form-urlencoded data, then return a 200 status response with a message. To do this, let’s create a new post route at the root by adding the following code to server.js:

router.post("/", async (req, res) => {
 
});

Now that we have our endpoint, we can also extract and view the text that has been sent by slack by adding the following line before we set the status:

const text = req.body.text;
console.log(`Input text: ${text}`);

For now, we’ll just pass this text into the response and send it back instantly, to ensure the slack app and function are communicating.

res.status(200);
res.send(text);

Now, when you type /faq <somequestion> on a slack channel, you should get back the same message from the slack slash command.

Formatting the response

Rather than just sending back plaintext, we can make use of Slack’s Block Kit to use specialised UI elements to improve the look of our answers. If you want to create a more complex layout, Slack provides a Block Kit builder to visually design your layout.

For now, we’re going to keep things simple, and just provide a response where each answer is separated by a divider. Add the following function to your server.js file after the post route:

const format = (answers) => {
 if (answers.length == 0) {
   answers = ["No answers found"];
 }
 
 let formatted = {
   blocks: [],
 };
 
 for (answer of answers) {
   formatted["blocks"].push({
     type: "divider",
   });
   formatted["blocks"].push({
     type: "section",
     text: {
       type: "mrkdwn",
       text: answer,
     },
   });
 }
 
 return formatted;
};

With this in place, we now need to pass our answers into this function, to format the answers before returning them to Slack. Update the following in the root post route:

let answers = text;
const formattedAnswers = format(answers);

Now when we enter the same command to the slash app, we should get back the same message, but this time in a formatted version!

Setting up Fauna

With our slack app in place, and a function to connect to it, we now need to start working on the database to store our answers. If you’ve never set up a database with FaunaDB before, there is some great documentation on how to quickly get started. A brief step-by-step overview for the database and collection is included below:

Create database

First, we’ll need to create a new database. After logging into the Fauna dashboard online, click New Database. Give your new database a name you’ll remember (I used “slack-faq”) and save the database.

Create collection

With this database in place, we now need a collection. Click the “New Collection” button that should appear on your dashboard, and give your collection a name (I used “faq”). The history days and TTL values can be left as their defaults, but you should ensure you don’t add a value to the TTL field, as we don’t want our documents to be removed automatically after a certain time.

Add question / answer documents

Now we have a database and collection in place, we can start adding some documents to it. Each document should follow the structure:

{
   question: "a question string",
   answer: "an answer string",
   qTokens: [
       "first token",
       "second token",
       "third token"
   ]
}

The qToken values should be key terms in the question, as we will use them for a tokenized search when we can’t match a question exactly. You can add as many qTokens as you like for each question. The more relevant the tokens are, the more accurate results will be. For example, if our question is “where are the bathrooms”, we should include the qTokens “bathroom”, “bathrooms”, “toilet”, “toilets” and any other terms you may think people will search for when trying to find information about a bathroom.

The questions I used to develop a proof of concept are as follows:

{
  question: "where is the lobby",
  answer: "On the third floor",
  qTokens: ["lobby", "reception"],
},
{
  question: "when is payday",
  answer: "On the first Monday of each month",
  qTokens: ["payday", "pay", "paid"],
},
{
  question: "when is lunch",
  answer: "Lunch break is *12 - 1pm*",
  qTokens: ["lunch", "break", "eat"],
},
{
  question: "where are the bathrooms",
  answer: "Next to the elevators on each floor",
  qTokens: ["toilet", "bathroom", "toilets", "bathrooms"],
},
{
  question: "when are my breaks",
  answer: "You can take a break whenever you want",
  qTokens: ["break", "breaks"],
}

Feel free to take this time to add as many documents as you like, and as many qTokens as you think each question needs, then we’ll move on to the next step.

Creating Indexes

With these questions in place, we will create two indexes to allow us to search the database. First, create an index called “answers_by_question”, selecting question as the term and answer as the value. This will allow us to search all answers by their associated question.

Then, create an index called “answers_by_qTokens”, selecting qTokens as the term and answer as the value. We will use this index to allow us to search through the qTokens of all items in the database.

Searching the database

To run a search in our database, we will do two things. First, we’ll run a search for an exact match to the question, so we can provide a single answer to the user. Second, if this search doesn’t find a result, we’ll do a search on the qTokens each answer has, returning any results that provide a match. We’ll use Fauna’s online shell to demonstrate and explain these queries, before using them in our function.

Exact Match

Before searching the tokens, we’ll test whether we can match the input question exactly, as this will allow for the best answer to what the user has asked. To search our questions, we will match against the “answers_by_question” index, then paginate our answers. Copy the following code into the online Fauna shell to see this in action:

q.Paginate(q.Match(q.Index("answers_by_question"), "where is the lobby"))

If you have a question matching the “where is the lobby” example above, you should see the expected answer of “On the third floor” as a result.

Searching the tokens

For cases where there is no exact match on the database, we will have to use our qTokens to find any relevant answers. For this, we will match against the “answers_by_qTokens” index we created and again paginate our answers. Copy the following into the online shell to see how this works:

q.Paginate(q.Match(q.Index("answers_by_qTokens"), "break"))

If you have any questions with the qToken “break” from the example questions, you should see all answers returned as a result.

Connect function to Fauna

We have our searches figured out, but currently we can only run them from the online shell. To use these in our function, there is some configuration required, as well as an update to our function’s code.

Function configuration

To connect to Fauna from our function, we will need to create a server key. From your database’s dashboard, select security in the left hand sidebar, and create a new key. Give your new key a name you will recognise, and ensure that the dropdown has Server selected, not Admin. Finally, once the key has been created, add the following code to server.js before the test route, replacing the <secretKey> value with the secret provided by Fauna.

const q = fauna.query;
const client = new fauna.Client({
 secret: "<secretKey>",
});

It would be preferred to store this key in an environment variable in Netlify, rather than directly in the code, but that is beyond the scope of this tutorial. If you would like to use environment variables, this Netlify post explains how to do so.

Update function code

To include our new search queries in the function, copy the following code into server.js after the post route:

const searchText = async (text) => {
 console.log("Beginning searchText");
 const answer = await client.query(
   q.Paginate(q.Match(q.Index("answers_by_question"), text))
 );
 console.log(`searchText response: ${answer.data}`);
 return answer.data;
};
 
const getTokenResponse = async (text) => {
 console.log("Beginning getTokenResponse");
 let answers = [];
 const questionTokens = text.split(/[ ]+/);
 console.log(`Tokens: ${questionTokens}`);
 for (token of questionTokens) {
   const tokenResponse = await client.query(
     q.Paginate(q.Match(q.Index("answers_by_qTokens"), text))
   );
   answers = [...answers, ...tokenResponse.data];
 }
 console.log(`Token answers: ${answers}`);
 return answers;
};

These functions replicate the same functionality as the queries we previously ran in the online Fauna shell, but now we can utilise them from our function.

Deploy to Netlify

Now the function is searching the database, the only thing left to do is put it on the cloud, rather than a local machine. To do this, we’ll be making use of a Netlify function deployed from a GitHub repository.

First things first, add a new repo on Github, and push your code to it. Once the code is there, go to Netlify and either sign up or log in using your Github profile. From the home page of Netlify, select “New site from git” to deploy a new site, using the repo you’ve just created in Github.

If you have never deployed a site in Netlify before, this post explains the process to deploy from git.

Ensure while you are creating the new site, that your build command is set to npm run build, to have Netlify build the function before deployment. The publish directory can be left blank, as we are only deploying a function, rather than any pages.

Netlify will now build and deploy your repo, generating a unique URL for the site deployment. We can use this base URL to access the test endpoint of our function from earlier, to ensure things are working.

The last thing to do is update the Slack endpoint to our new URL! Navigate to your app, then select ‘slash commands’ in the left sidebar. Click on the pencil icon to edit the slash command and paste in the new URL for the function. Finally, you can use your new slash command in any authorised Slack channels!

Conclusion

There you have it, an entirely serverless, functional slack slash command. We have used FaunaDB to store our answers and connected to it through a Netlify function. Also, by using Express, we have the flexibility to add further endpoints to the function for adding new questions, or anything else you can think up to further extend this project! Hopefully now, instead of waiting around for someone to answer your questions, you can just use /faq and get the answer instantly!

---

Matthew Williams is a software engineer from Melbourne, Australia who believes the future of technology is serverless. If you’re interested in more from him, check out his Medium articles, or his GitHub repos.

---

The post Create an FAQ Slack app with Netlify functions and FaunaDB appeared first on CSS-Tricks.

You can support CSS-Tricks by being an MVP Supporter.