disclaimer: My thoughts, ideas, are half-baked. I do not wish to conform or convince the reader, as I have no rights to plunge you into the deep with me. I am but a feeble fellow, and I practice my art in my own way narrowed down with curiosity and perseverance.

Introduction

This paper is not written to start another episode of a flat or round earth debate. Rather, I aim to spark a thought, and discuss what abstraction is, the pyramid of abstraction, and the price for abstraction.

Wikipedia defines Abstraction as a process wherein general rules and concepts are derived from using and classifying specific examples, literal (real or concrete) signifiers, first principles, or other methods. It also says it is the outcome of a concept that acts as a common noun for all subordinate concepts and connects any related concepts as a group, field, or category.

I had initially thought about writing this blog targeted at the software - tech-folks. But I implore you to walk with me, I will make this relatable to both code and real life. [you might find computer-related terms, abeg no vex]

 My thoughts on this topic began to take form after reading the book “A History of Western Philosophy” by Bertrand Russel. On advancing what Western civilization has taught us, and where we are now. For a moment, I urge you to pause and think, from the first men, who wrote on stones with carvings, to the first paper material “papyrus”, down to our present use of computers. it can be seen clearly how the art of writing has evolved. Ask yourself, has this new form introduced a new set of super writers, or made the tradeoffs favorable? What would have been the narratives for writers like William Shakespeare, or for men like Socrates who was largely fond of teachings in the Greek square, perhaps if X existed in his time?

I like to think of abstractions as introducing new layers to a process or structure. Hidden and complicated facets can be simplified into something smaller, isolating away the important machining part of the system. For instance, should the final product of making a pen require 10 steps, it can be abstracted to require 4 - 6 steps. The easiest way to relate to abstraction can be likened to an artist or a painter. To draw the human body, you’d probably start with the head down to the legs. But with abstraction, a faster method can be introduced involving the use of certain shapes, requiring less time but with equal results.

At this point, it would resonate with you that the mind forces us to abstract with whatever we find complex. Perhaps this approach has done more damage than good, how then can we avoid the wrong abstraction? The Wrong Abstraction by Sandi Metz highlights that

duplication is far cheaper than the wrong abstraction.

The Abstraction Pyramid

The Abstraction pyramid defines the fundamental concepts on which ideas and logic evolve. The main idea, I believe is that every work should be based on concrete details that can spawn into a subset of layers at the top. For instance, a piece of text or structure that is too abstract could rather be difficult to connect with, or even hard to understand. Physical details like smell, touch, sight, or even sound evoke clear thoughts aiding the connection of abstracted ideas from conceptual beliefs or facts.

The abstraction pyramid can effectively explain how complex systems or base ideas are built upon. With each layer, an abstraction of the other. The higher the pyramid, the higher the number of simpler interfaces. For clarity, the pyramid can be divided into three subsets.

• The Base Layer: Represents the primary systemic beliefs on which systems are built. They could be laws of nature or primary instincts. Newton’s fundamental ideas on the laws of Motion evolved into several ways we can interact with machines or navigate the universe. From building bridges to space travel. These ideas hold the fundamental laws we obey or put into practice.

• The Middle Layer: This layer is birthed one abstraction away from the base layer. This layer serves to bridge the gap that connects the top and the base layer. More noticeable attributes include a step further into complexity, domain expansion, and newer ideologies. Each of these attributes hides the underlying truth from the surface. In Biology, tissues, and organs can be categorized as known examples of the middle layer. Socio-political systems, involve communities or institutions.

• The Top Layer: represents the most abstract of complex of systems. Here, there’s a perceived high-level functionality that leverages the base layer streamlined to serve a single purpose. In his layer, behaviors can be predictable. They integrate to form cohesive bonds with the middle layer. Much more likened to non-experts or end users to interact with, because of its known simplified interface.

The price for Abstraction

Abstractions are never free, there’s always a price attached. The creator of the C++ programming language Bjarne Stroustrup: In his book Foundations of C++ said:

In general, c++ implementations obey the zero-overhead principle: What you don’t use, you don’t pay for. Furthermore, what you do use, you couldn’t hand code any better.

To properly put into perspective what Bjarne meant, let’s go with this approach: When you visit the market, you only pay for what you have in your bag or choose to go home with. You don’t have to pay for what you see, that you didn’t take. So as much as in C++, what you don’t use, doesn’t add up for extra weights, or spikes in low system performance. Your program only compiles with what you’ve used, with no need to do it any better. You see, this is not the same for JavaScript, like so.

Here’s a simple example: A web server comparison in C++ and JavaScript, that implements a simple POST method to put items in a cart.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// c++ example using crow framework

#include "crow.h"
#include <vector>
#include <string>
#include <chrono>

int main(){
 crow::SimpleApp app;
 std::vector<std::string> cart;

 CROW_ROUTE(app, "/add")
 .methods("POST"_method)
 ([&](const crow::request& req){
  // neglect all err handling and so
  auto start = std::chrono::high_resolution_clock::now();
  auto body = crow::json::load(req.body);
  
  std::string item = body["item"].s();
  cart.push_back(std::move(item));

  auto end = std::chrono::high_resolution_clock::now();
  std:chrono::duration<double, std::milli> duration = end - start;
  
  crow::json::wvalue res;
  res["message"] = "successful";
  res["cart_size"] = cart.size();
  res["time_taken"] = duration.count();
  
  return crow::response(res);
 });

 app.port(3030).run();
 
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

// JavaScript example using Express

const express = require("express")
const app = express()
app.use(express.json())

let cart = []

app.post('/add', (req, res) => {
 // neglect all err handling and so
 const start = process.hrtime()
 
 const item = req.body.item
 
 cart.push(item)
 const end = process.hrtime(start)
 // convert the time to milliseconds
 const duration = (end[0] * 1e9 + end[1]) / 1e6 

 res.json({
  message: 'successful',
  cart_size: cart.length, 
  time_taken: duration
 })
})

app.listen(3000, () => console.log('Server running on port 3000'));

Now, let’s break down what both code does, and the adoption of abstraction therein. Comparing how C++ or JavaScript might handle this task.

The code runs a simple web server in port 3030 and 3000 respectively. In the request made, the item is stored in an array, and the time taken is calculated with a response returned. Software engineers are familiar with high, low-level languages, and machine code, etc. Now let’s look at this code and break it down in terms of memory management, performance, type safety, and concurrency. Recall I made attributes of the middle and top layers, and how they can be subdivided into different facets. In real-life practice, the performance or bottlenecks might not be noticeable. However, as it grows to handle more users/requests, and other complex operations one language stands out from the other. From these two code examples, the key differences are:

• C++ programs are directly compiled to run directly on the operating system, allowing for more direct access to the system’s hardware.

• On the other hand, JavaScript is interpreted or JIT(Just In Time) compiled within a runtime environment(v8 Engine, Event Loop, Garbage Collector). Which is usually the browser, providing another layer of abstraction away from the hardware.
  Further examples for the price of abstraction in computer science

The lack of abstraction can be expensive and time-consuming. You can try writing some software in Assembly to find out, or to bankers who depend on abstracted risk models for each loan application in detail.

I’ve spent more time in my life learning that we can always use abstractions but at a price. The greater good here is finding one whose values align well with your interests, or what you aim to achieve. This is not to disapprove those who prefer to build from scratch, a wise and thoughtful approach I must commend you. I came across something on Hacker News that says, “People tend to discount the cost of all abstractions that they have internalized. Therefore when they get to a new environment, they immediately try to recreate and incorporate the abstractions that they have previously used”. This largely translates to, people taking up abstractions to prevent themselves from running into bigger problems in the future, an alternate route with no effects, whose path should be taken all the time.

Conclusion

Abstractions evolve with time, like a parasite. It grows and studies the host environment. What would you call a good or bad abstraction, is it one which allows you to look under the hood, allowing for better optimization? That is a question only you can answer my friend. I am but a messenger, like Morpheus in the Matrix movie. I present to you the red pill, which enables you to understand what is happening inside the abstraction, or the blue pill, allowing you to only use that abstraction.

Aloha,
David❤️

TLDR

• The price of abstraction (repec.org)
• Abstraction (computer science) - Wikipedia
• Abstractions, Idealizations, and Evolutionary Biology Peter Godfrey-Smith https://petergodfreysmith.com › PGSAbstractnId...
• The Evolution of Abstraction in Programming Languages - DTIC Defense Technical Information Center (.mil) https://apps.dtic.mil › sti › pdf › ADA059394

Dear reader, if you’re thinking this is a horrible title for such a read, I am writing to tell you that I can’t help it. You’d have to pardon me, just incase.

June - July 2022

I love to make plans a lot, every single day of my life and every event that occurs. I can’t always help it, but jot down these little task on my journal, notebook or even on my mind.

In as much as I love to keep things in my life private, I’d share my July experience with you. I started the year as an intern in my university, working with my professor as a researcher, well yeah. It was fun, and a position where I learnt new and interesting topics every single day. It has been that way until the strike that affected Nigerian Universities, and somehow looks like it isn’t coming off soon, had to put me out of work, for most of the time.
If you’re reading this, must have noticed that I am a software engineer. I do mostly backend stuff with Java (springboot/spring frameworks) and Node.js(express).

The Job Experience

The experience has been a bitter sweet type. I kid you not, whosoever is job searching and finally lands one, should be offered a therapy session to ease out from the stress and anxiety.

I’ve sent in several applications, open-source contributions, community events and the likes and yet there’s nothing. Trust me, I had never had this experience in my life before.

On the positive side, I learnt a lot about crafting a near perfect resume, writing cover letters and getting out of my comfort zone to reach out to people. So far, it has been a good one, but the pain is too much.

Looking back to how I started the year, and where I am now, I can say I’m excited but not too, about my growth.

Stuffs I’ve Leant

I’ll be sharing this in no order, as I can’t wrap my head write now. Writing this post on my laptop.
I’ll try to elaborate in the nest way I can for certain topics, or habits.

• Next.js/React

Yes, I can boldly say this is one thing that July has blessed me with. I never enjoyed doing frontend development in the beginning. In fact, I hated the thought of building frontend applications. Not until I was forced to build my online portfolio. That alone was another hurdle in choosing the right framework, hosting, and what contents I should upload on there.

Settled for Next.js after doing my research. But most importantly, cos a senior friend I knew wanted me to help out with a project he had in hand at te time.

I was debating between writing next.js with javascript or with Typescript. See, the only time I laid my hands on Typescript was with angular, and that was about 4 months ago, didn’t know shit about what I was doing. I just looked at docs and built my frontend application I was testing for my full-stack project.

I had to settle for Javascript, but looking at how much I’ve carried on my head, I can comfortably pickup Typescript where necessary.

• Learnt how to write optimal, standard Next.js application for production.

• How to setup dev environments to ensure strict software development cycles amongst developer teams.

• Getting familiar with the Next.js ecosystem

• Understand SSR with Next.js

• How Next.js works under the hood

I’d love to do a quick recap of how next.js works, and how you get your contents displayed.

Before I continue, in my experience I think the best way to host next.js applications should be with Vercel, AWS works as much that is if your team/you host on AWS. But it’s really easy with Vercel and vercel fully optimizes for Next.js.

When your Next.js project is hosted, and a client wants to access the content on the page. Here’s a working procedure:

• User visits your website
• The server then proceeds to generate a rendered HTML file that fetches all of the required data. || site displays no content, nothing is visible.
• The browser then renders your site and downloads all your js files || website is visible but ui is not interactive.
• Browser then executes the your js files, and your application is visible and ready for use by the client to interact with.

Built my blog and portfolio using Next.js and tailwindcss which is now my favorite css framework.

• Java

Got back to reading Java books, two of which I haven’t completed. Practiced more data structures and algorithms, learnt more about core Java, it’s API’s and system designs. Getting my hands more on cloud tools, CI/CD. Likes of AWS Kubernetes, Docker, Jenkins, and Kafka.

Allow me to stop here, thank you.

Aloha❣
David

Typesense offers a number of options to integrate with different tech stacks, and you may have noticed that both internal and external communities are working around the clock to make that a reality. Numerous use cases can also be found in your preferred language or framework of your choice. The integration of Typesense with Laradock and Laravel Scout will be covered in this particular article.

If you are reading this and still unfamiliar with Typesense, here’s a short intro:

A modern, privacy-friendly and open source search engine built from the ground up using cutting-edge search algorithms, that take advantage of the latest advances in hardware capabilities.

You can find additional documentation and a proper guide on how to get started here. Still unsure about Typesense against other search engines, find a detailed comparison here.

Laradocks vs Laravel Scout

For the non-PHP devs, I’ll briefly explain what Laradock and Laravel scout is before I dive deeper.

Laradock is a PHP development environment for docker that comes with a pre-configured service for common services. You can find a long list of services that it supports, and you can have them turned on/off, or as many instances as you want.

Laravel Scout is a PHP library offering driver-based solutions to handle manipulation with the index data model by adding a full-text search for that model.

Laravel Scout supports MySQL and PostgreSQL databases and comes with Algolia or MeiliSearch. During development, a collection driver serves locally and does not require third-party integration or dependencies.

Setting Up Typesense with Laravel Scout

Create a new PHP application by running one of the code commands mentioned below:

1. Using Composer create-project

1

composer create-project laravel/laravel typesense-demo

2.Using the Laravel Installer

1

laravel new typesense-demo

The above command creates a new Laravel project in your specified directory

To begin a new Laravel project, I’ll use the command

1

composer create-project

Installing Laravel scout with the composer command

1

composer require laravel/scout

Before installing the Typesense driver, install the correct HTTP plug adapter based on the guzzlehttp/guzzle version you have available on your system.

i.e if you are running on Laravel 8, the guzzlehttp/guzzle version will be version 7.

Typesense uses this HTTP plug to communicate with various PHP HTTP libraries through a single API.

Here’s the command

1

composer require php-http/guzzle7-adapter

Install the Typesense driver

1

composer require typesense/laravel-scout-typesense-driver

Initiate the service provider

1
2
3
4
5

// config/app.php
'providers' => [
    // ...
    Typesense\LaravelTypesense\TypesenseServiceProvider::class,
],

Add Laravel scout as a provider to avoid the “unresolvable dependency error”.

1
2
3
4
5

// config/app.php
'providers' => [
    // ...
    Laravel\Scout\ScoutServiceProvider::class,
],

Configure your environmental variables inside the .env file.

Add SCOUT_DRIVER=typesense to the .env file.

Publish your configuration for Laravel scout with the command

php artisan vendor:publish --provider="Laravel\Scout\ScoutServiceProvider"

This command enables us to modify the created config/scout.php file and copy the configuration file in the application from the vendor package file in our case, the typesense-driver and all tags are published. Thus, any changes made to the code base will not be lost. The publish command also provides a few options that specifically assist in choosing which file should be published.

Add the following code within config/scout.php

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// add to config/scout.php

'typesense' => [
    'api_key'         => 'abcd',
    'nodes'           => [
      [
        'host'     => 'localhost',
        'port'     => '8108',
        'path'     => '',
        'protocol' => 'http',
      ],
    ],
    'nearest_node'    => [
        'host'     => 'localhost',
        'port'     => '8108',
        'path'     => '',
        'protocol' => 'http',
    ],
    'connection_timeout_seconds'   => 2,
    'healthcheck_interval_seconds' => 30,    
    'num_retries'                  => 3,
    'retry_interval_seconds'       => 1,
  ],

Now we have completely configured our Typesense integration with Laravel scout.

For the primary usage, see the following code below or you can consider checking the Laravel Scout documentation first.

Let’s add a searchable trait to the models you want to perform a search on. toSearchableArray method, in this case, defines the fields that the model should make a search on and implements TypesenseSearch.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
use Typesense\LaravelTypesense\Interfaces\TypesenseDocument;
use Laravel\Scout\Searchable;

class Todo extends Model implements TypesenseDocument
{
    use Searchable;
      /**
     * Get the indexable data array for the model.
     * @return array
     */
    public function toSearchableArray()
    {
        return array_merge(
            $this->toArray(), 
            [
 // Cast id to string and turn created_at into an int32 timestamp
 // in order to maintain compatibility with the Typesense index definition below
                'id' => (string) $this->id,
                'created_at' => $this->created_at->timestamp,
            ]
        );
    }
     /**
     * The Typesense schema to be created.
     * @return array
     */
    public function getCollectionSchema(): array {
        return [
            'name' => $this->searchableAs(),
            'fields' => [
                [
                    'name' => 'id',
                    'type' => 'string',
                ],
                [
                    'name' => 'name',
                    'type' => 'string',
                ],
                [
                    'name' => 'created_at',
                    'type' => 'int64',
                ],
            ],
            'default_sorting_field' => 'created_at',
        ];
    }
     /**
     * The fields to be queried against. See https://typesense.org/docs/0.21.0/api/documents.html#search.
     * @return array
     */
    public function typesenseQueryBy(): array {
        return [
            'name',
        ];
    }    
}

You can now sync your data with search services:

1

php artisan scout:import App\\Models\\Todo

This command now allows you to get your model data synced with the search services, it also comes with automatic data syncing for any changes you make.

Perform a search on your models with

1

Todo::search('Test')->get();

Add records via query

1
2
3
4

$todo = Todo::find(1);
$todo->searchable();
$todos = Todo::where('created_at', '<', now())->get();
$todos->searchable();

Again the searchable() method here groups the result of your query and then proceeds to add the records to your search index.

Conclusion

Since Typesense offers out-of-box features with several language options for an open-source search engine compared to its competitors, it is worth looking into and stands out from the rest.

This proves a massive and active community behind it, continually building tools and integrations for software development.

TLDR

1. Typesense API reference: A helpful section that includes all the documentation details on all API endpoints available on Typesense and how you can use them.

2. PHP - Typesense Search Application: Create a book search application with the typesense client

3. Typesense: Setup Typesense locally or with cloud

4. Typesense Github: Source repo for Typesense on Github.

5. Laradock installation: Setting up Laradock with docker plus official documentation.

6. Laravel scout: Official documentation for Laravel scout.

7. Laravel-scout-typesense-driver: Github repo for Laravel scout Typesense driver.

8. Laravel: Getting started with Laravel.

I know It’s been months since I last made an update here, won’t happen anymore, I really hope that this turns out to be a weekly issue where you get updates.
I’ll do all that’s within my power, salut!😎😉

PS: I have some exciting topics saved on my drafts, so yeah I’ve been cooking…

Let’s talk about spring boot and the latest official release for graphQL 1.0. For the non-tech folks, no vex. But I’d try my best to explain this in layman’s terms, so let’s go…

GraphQL release became official following the recent release of springboot 2.7.0 see image attached below.

A brief introduction to what graphQL is, and what I think you should know to get started.

GRAPHQL 101

GraphQL is a web query language that is used to build web APIs Application Programming Interface.
The most common way to do this was with REST API(Representational State Transfer), but the increase in demand for more data in building a web application meant an increase for more endpoints to feed to the user, from third-party services, IoT devices, microservice, and mobile applications all requiring a specific kind of data. No matter the complexity of these applications, they were often limitations. The data sets needed on different applications are usually not the same, some required priority on faster connections, others required robust data to process information in real time, they were so many restraints to be concerned about.

Then came the official release of GraphQL to the public by Facebook now META in 2015. It was introduced to at least solve this issue. You can execute a specific query command through your endpoints to get the specific kind of data needed for your applications. These meant that services could fetch just the right amount of data, from a single query.

GraphQL Type System

Okay, we’re now one step closer to building our application. Having an understanding about the graphql type system will help you in building your graphql schema, which is responsible for communicating with client requests when a query is executed.

An example look at the schema created for an employee web API.

The employee type has an ID, firstName, lastName, email, homeAddress, and phoneNumber. In this case through a single query, data can be fetched to get all employees by firstName, or even email.

The graphql schema is basically the same that is written across multiple programming languages, similar to JSON(Javascript Object Notation). You can read the schema structure in a Java application in the same way, for a Javascript, Go or C++ application.

In setting up your application, you can decide what object types and fields you want present in your application. Your object “type“ is synonymous to the model that is created for your application. In the image above, I created a model that contains Employees (id, firstName, lastName, email, homeAddress, and phoneNumber). The values on the right defines the parameters that should be inputed (string, boolean, int: recall data types), while the values on the left represents the data field names. ! attached to a value type indicates that it should not return a null response upon query.

Operation Types

• Query: Used to read data. Practically performs the same function similar to the GET method in a REST API. In spring boot you use the annotation @QueryMapping.

• Mutation: Used to perform Delete, Update and Create data similar to PUT, POST and DELETE methods in REST API. In spring boot you use the annotation @MutationMapping.

• Subscription: Likely performs the same function as the Query method to get data, but in this case is used for operations that their results tends to change over time given for a certain period.

Building our Server

Now let’s take a look on how to build our web server. For the sake of this demo purpose I’ll try to not make this a boring and long read, I’ll just add snipets of my code samples which I think are important and helpful. I already did a ton of explanation which should get your started and I will assume you are already familiar with Java and Spring boot or building web APIs.

If you aren’t, worry not. I’ll share some of the best materials I’ve come across on the web at the later ending of this post. If you need to see my working code, make a request and I’ll drop a link to the github repo.

Dependencies you need to get started, here’s what my pom.xml looks like.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-graphql</artifactId>
</dependency>
<dependency>
   <groupId>com.graphql-java</groupId>
   <artifactId>graphiql-spring-boot-starter</artifactId>
   <version>5.0.2</version>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>com.h2database</groupId>
   <artifactId>h2</artifactId>
   <scope>compile</scope>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-webflux</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.springframework.graphql</groupId>
   <artifactId>spring-graphql-test</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.projectlombok</groupId>
   <artifactId>lombok</artifactId>
   <version>1.18.24</version>
   <scope>provided</scope>
</dependency>

PS:
You’ll need to add the spring graphiql dependency. The graphiql dependency is a UI companion for your graphql server to perform query operations and test your APIs on your browser, no need for a HTTP client.

1
2
3
4
5

<dependency>
   <groupId>com.graphql-java</groupId>
   <artifactId>graphiql-spring-boot-starter</artifactId>
   <version>5.0.2</version>
</dependency>

In your application.yml also lookout for the spring-graphql-graphiql-enabled = true to enable the graphiql browser UI.

1
2
3
4

spring:
  graphql:
    graphiql:
      enabled: true

I used the H2 database and spring data jpa just for this demo application. In a real world project, you can choose to use an SQL or No-SQL database.

My application.yml file.

My controller class

Graphql query results

TL;DR

Important Resources

• Graphql Resources
• Getting started with graphql
• Spring Rest Services
• Java Roadmap
• Use the roadmap as a guide on how to start, find free courses online.

PS: Anticipate an issue about structured list plus resources to learn java, and how I started my java journey.

Thank you for reading, and till next time….

Don’t forget to like and comment!

Aloha ❤

David.

In software development, creating data backups is an important and safe practice in the software development cycle process. The data is typically stored on the same device or cloud storage. This system approach is used in the software process to restore the system or data in a particular directory at a specific time to any previous state of the system.

Typsense creates data backup by means of snapshots that can be implemented with a single API call. Cluster operations can be utilized for these backup processes. Snapshot backups define where and how your data is stored, or organized. cluster backups can be created from snapshots from a Typesense node state and the data in the specified directory can be really efficient for rapid recovery as a data directory when it is needed.

Why backup is needed

With a wide variety of methods or programs that offer support for data backup, let’s cover in a few lines why data backup is essential for developer teams.

1. Protect important data from loss in the occurrence of a data breach, or physical disaster.

2. Improve software development cycle as data can be recovered from different individual team members when needed.

3. Increase software team security from loss of data due to data breach.

4. Quickly backup and restore data/system in different cycle states.

5. But before we execute any backup operations, let’s first create our Typesense server step by step and then demonstrate how to perform a backup using javascript.

Starting up Typesense Server

Let’s get started by starting our Typesense server indexing dummy content/data for demo purposes only. We must first set up and start our Typesense server before we can begin creating our Typesense index. Let’s make a script/ command to configure our docker image and run the docker container that will basically bind and connect to where we’ll be keeping the data.

Let’s start by making a new script. So create an empty project directory and a scripts folder inside it, because we’ll have a few scripts loaded in here that will allow us to index the data and start the typesense server, or the docker container, but first we need to set up and initialized our npm and install a few dependencies.

Initializing npm

1

npm init -y

Once you’ve initialized npm, it’s time to install and update some packages. For now, we’ll add the Typesense package only.

1

npm i typesense

Your package file should look like this once you install the typesense package.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  "name": "typesense-databackup",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typesense": "^1.2.2"
  }
}

Setting up Server run script

Inside the scripts folder, make a new file called runServer.js. Despite the fact that this script would just run the docker command, we must first configure it. The folder structure should look like this.

The very first step is to use the node.js child process library to run and import exec commands.

1
2

//scripts/runServer.js
const { exec } = require("child_process");

Let’s use our own script command to start our typesense server. To start the docker in the background, we’ll use the “docker -run” and “docker detach” commands, and then provide the port we want it to utilize. The volume section is the most crucial part of this command; we’re binding the volume, which is essentially a mechanism of saving the typesense’s data, which is usually saved inside the container. As a result, typesense will save the data in the container’s /data folder, then expose and connect that folder to the system location we designate.

Following that, we’ll specify the image we want to run inside our docker container, then the data directory and the api key, which you can fill with whatever you want, and then configure listen port, and finally enable cors to avoid any CORS related issues.

1

const command = `docker run -d -p 8108:8108 -v/tmp/typesense-server-data/:/data \ typesense/typesense:0.22.2 --data-dir /data --api-key=typesenseDatabackup --listen-port 8108 --enable-cors`;

Finally, run the program and include an error handler in case anything goes wrong. Hence, your runServer.js code should look like this:

1
2
3
4
5
6
7
8
9

// scripts/runServer.js
const { exec } = require("child_process");
const command = `docker run -d -p 8108:8108 -v/tmp/typesense-server-data/:/data \ typesense/typesense:0.22.2 --data-dir /data --api-key=typesenseDatabackup --listen-port 8108 --enable-cors`;
exec(command, (err) => {
  if (!err) console.log("Typesense Server is running...");
  if (err) {
    console.log("Error running server: ", err);
  }
});

Now that we have our runServer.js script ready to go, we can simply update the script tag in our package.json file. So that whenever you want to start the server, you simply open a command prompt and run the npm command to fire up the entire script.

1

"start": "node scripts/runServer.js"

Your package.json file should look like this after you’ve configured the script tag.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  {
  "name": "typesense-databackup",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node scripts/runServer.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typesense": "^1.2.2"
  }
}

Finally, type npm run start to start the server, which will start your docker, allowing you to move on to the next stage of indexing the data/collection.

Note: Before you even run this script make sure your docker desktop is up and running.

The next step is to actually populate or index the data inside the typesense server, so now that we have the server up and running, we’ll be populating a simple dummy dataset inside the typesense server so that we can query it and conduct a backup later on. So let’s get to work on the data-import script. We’ll begin by creating an dataIndexer.js file in the scripts folder we previously built. Once you do that your folder structure should resemble something like this.

Indexing data on typesense: a step-by-step guide

We must first add and import the Typesense library to our project.

1
2

// dataIndexer.js
const Typesense = require("typesense");

Let’s construct a self-executing function that will execute every time the script is invoked, and make it asynchronous so we can leverage the async await functionality. Simply create module.export and export the self-executing function inside it as asynchronous so that we can write a script that reads the data and collects it, manages the typesense server, and indexes it. So, first and foremost, we must set up the typesense client in order to connect to the server and begin managing, indexing, and retrieving data. Create a typesense config variable and pass the properties of the node to it.

so this specific nodes is actually an array that holds the actual configuration for each server that you want to connect into and that you want the client to access, so currently, we only have one server running so we are going to utilize only one nodes. Next, inside the nodes array, supply the host, typesense port, and protocol it uses, as well as the API key.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// dataIndexer.js
const Typesense = require("typesense");

module.exports = (async () => {
 const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };
})();

Let’s utilize the typesense configuration to make a Typesense client.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// dataIndexer.js
const Typesense = require("typesense");
module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
     {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };
  console.log("Config: ", TYPESENSE_CONFIG);
  const typesense = new Typesense.Client(TYPESENSE_CONFIG);
})();

This is an important stage because it is here that we offer the schema that will be used to index our data into the actual typesense database, therefore schema is very important. It has a simple syntax and is easy to work with. Schema is essentially a description of how your data will be saved. We have a title, synopsis, genre, and popularity for our schema. So you only want to put the fields you want to index in schema. Begin by naming the schema and ensuring that the number of documents is zero.

Next, supply the name, type, and facet for the fields, which will be an array of objects containing every single field that we want to index and save in our database. If you’re wondering what a facet is, it’s a feature that allows you to create categories based on a subset of qualities so that users may refine their search results. This is the way your schema should appear.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

// dataIndexer.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "dummyData",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };
})();

So, if you want to index all of the data in the array of genres, for example, you’ll need to store each level of the array in their specific/own field.

Now let’s read the movies from the JSON files and import the dataset. It’s time to launch the Typesense client and connect to a schema within it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

// dataIndexer.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "dummyData",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };

  const dummyDataSet = require("../dataset/dummyData.json");

  try {
    const collection = await typesense.collections("dummyData").retrieve();
    console.log("Found existing collection of dummy Data");
    console.log(JSON.stringify(collection, null, 2));
  } catch (err) {
    console.error(err);
  }
})();

If a data duplication issue arises during loading the data, simply add the following bit of code to the dataIndexer.js file before constructing the schema since it will simply remove the existing data and replace it with the new ones.

1
2
3
4
5

if (collection.num_documents !== dummyDataSet.length) {
     console.log("Collection has diff number of docs than data");
     console.log("Deleting collection");
     await typesense.collections("dummyData").delete();
 }

The final step is to create a dummyData collection. A Collection in Typesense is a group of connected Documents that works like a table in a relational database. When we establish a collection, we give it a name and explain the fields that will be indexed when a document is added to it.

Your final code inside dataIndexer.js the file should look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

// dataIndexer.js
const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };

  console.log("Config: ", TYPESENSE_CONFIG);

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const schema = {
    name: "dummyData",
    num_documents: 0,
    fields: [
      {
        name: "title",
        type: "string",
        facet: false,
      },
      {
        name: "synopsis",
        type: "string",
        facet: false,
      },
      {
        name: "genre",
        type: "auto",
        facet: true,
      },
      {
        name: "popularity",
        type: "float",
        facet: true,
      },
    ],
    default_sorting_field: "popularity",
  };

  const dummyDataSet = require("../dataset/dummyData.json");

  try {
    const collection = await typesense.collections("dummyData").retrieve();
    console.log("Found existing collection of dummy Data");
    console.log(JSON.stringify(collection, null, 2));

    if (collection.num_documents !== dummyDataSet.length) {
      console.log("Collection has diff number of docs than data");
      console.log("Deleting collection");
      await typesense.collections("dummyData").delete();
    }
  } catch (err) {
    console.error(err);
  }

  console.log("Creating schema...");
  console.log(JSON.stringify(schema, null, 2));

  await typesense.collections().create(schema);

  console.log("Populating collection data...");

  try {
    const returnData = await typesense
      .collections("dummyData")
      .documents()
      .import(dummyDataSet);

    console.log("Return data: ", returnData);
  } catch (err) {
    console.error(err);
  }
})();

We can simply update the script tag in our package.json file now that our dataIndexer.js script is ready to use.

Finally, your package.json file should look like this once you’ve updated the script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

//package.json
{
  "name": "typesense-databackup",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node scripts/runServer.js",
    "indexer": "node scripts/dataIndexer.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typesense": "^1.2.2"
  }
}

Finally, start indexing the data in typesense server by entering the following command, and your data will begin to populate within the typesense server; however, make sure your dataset is ready before you do so.

1

npm run indexer

Now that we’ve loaded our Typesense server with data, we can use it to execute data backup.

How Typesnse backups works

When performing multiple snapshot backups, logs are registered thereby allowing in real-time, the data saved to be recalled and backed for recovery. This process offers simple and quick access to your data with real-time responses affirming your backed-up data.

Faster transfers of snapshot data can be transferred or copied into multi-regional clusters.

How to create a snapshot for backup

Create and back up a snapshot file, that can be restored as a data directory.

Now that our Typesense server is up and running and we’ve indexed our data, it’s time to create a data backup. To do so, all you have to do is perform a snapshot operation with Typesense, which is only one line of code, so let’s get started with an example. For this example, we’ll use javascript, but note that you can perform this operation in a variety of programming stacks, as shown in the documentation.

To begin, go to your script directory and create a file called databackup.js. Once you’ve done that, your folder structure should resemble like this.

We’ll write practically identical code within it, so first we’ll create a typesense connection to the server, then we’ll use the typesense settings to construct a Typesense client, and finally we’ll produce a snapshot and save it in the temporary directory path.

This is how your javascript code should look to perform a snapshot in Typesense.

1

Client.operations.perform('snapshot', {'snapshot_path': '/tmp/typesense-data-snapshot'})

Your final code should look somewhat like this once you’ve configured your snapshot to your dataBackup.js file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

const Typesense = require("typesense");

module.exports = (async () => {
  const TYPESENSE_CONFIG = {
    nodes: [
      {
        host: "localhost",
        port: "8108",
        protocol: "http",
      },
    ],
    apiKey: "typesenseDatabackup",
  };

  const typesense = new Typesense.Client(TYPESENSE_CONFIG);

  const backupData = await typesense.operations.perform('snapshot', {'snapshot_path': '/tmp/typesense-data-snapshot'})

  console.log(JSON.stringify(backupData, null, 2));

})();

Once you’ve finished with the code, make a separate script to run our command. When you’re finished, your package.js file should look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

{
  "name": "typesense-databackup",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node scripts/runServer.js",
    "indexer": "node scripts/dataIndexer.js",
    "backup": "node scripts/dataBackup.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typesense": "^1.2.2"
  }
}

Finally, start your data backup snapshot operation in typesense by entering the following command in your terminal.

1

npm run backup

You should receive the following response after running the command above.

1
2
3

{
  "success": true
}

Boom! Our data backup was successful, as you can see in the screenshot below.

This was a JavaScript example, but below is a list of backup operations you can try out with various programming language stacks, so give it a shot and see how easy it is to do data backup snapshot operations in Typesense.

1
2
3
4

HashMap<String, String> query = new HashMap<>();
query.put("snapshot_path","/tmp/typesense-data-snapshot");

client.operations.perform("snapshot",query);

1

client.operations.perform('snapshot', {'snapshot_path': '/tmp/typesense-data-snapshot'})

1

$client->operations->perform("snapshot", ["snapshot_path" => "/tmp/typesense-data-snapshot"]);

A JSON response is returned upon successful creation

PS: snapshot_path indicates the working directory on the Typesense server where the snapshot should be saved and is required for every data backup operation.

To restore your previous backups, start Typesense on a single node cluster, using the data directory backed up to set the lead node. Subsequently, you can add follower nodes by updating the nodes file.

Conclusion

One cool feature about snapshots is that they can be created in a matter of seconds and as often as is needed, as a consistent process during the software development cycle. In this post, we have talked about why data backups are important, how Typesense backups works, and how to create your own snapshot backup.

On a good note, Typsense has a robust list of API resources for your support, plus other official documentation that you will find useful.

Interested in contributing to open-source, find other open-source projects here, or register to start your first contributor journey.

Find out more about Aviyel, a community-driven monetization platform for your open source projects.

TL;DR

1. Cluster operations

2. Typesense GitHub

3. Download Typesense

4. Typesense guide