What is Deno?

Oversimplifying, we can say that Deno is an alternative to Node.js.

More precisely, we can read the following description on Deno's website:

Deno is a simple, modern and secure runtime for JavaScript and TypeScript that uses V8 and is built in Rust.

Let's delve deeper into three key characteristics that distinguish Deno from Node.js:

• Simple: as we will soon see, Deno is simple to install, being shipped as a single executable file; Deno is also simple to use for small and larger scripts thanks to its powerful CLI, modern features and standard library; finally, learning Deno is also simple thanks to its short and execellent manual

• Modern: Deno is built with modern technologies and follows a modern philosophy; it natively supports TypeScript and ES modules; it doesn't require a centralized package management solution; it also provides useful developer tools out of the box such as an opinionated source code formatter and a testing framework

• Secure: by default, Deno prevents access to critical resources such as files, network connections and environment variables; using these resources requires an explicit permission grant by the user

Now that we have got to know Deno better, let's start using it.

What are we building?

We are going to write a script that reads the stream of comments posted to Hacker News in real time and prints them in the console.

We will use TypeScript to write the script and Deno to run it; prior experience in TypeScript or Deno is not required since we will go step by step.

I assume that you are using a Linux distribution where a text editor and a terminal are available, however you should be able to follow the same instructions on Windows or Mac with minimal differences.

When following along, you can refer to the source code for the script, integrating it with the Deno manual and the TypeScript Handbook if necessary.

Setup

Let's start by installing Deno on our machine and verifying that it works.

According to Deno's installation instructions, we need to run the following command in our terminal:

curl -fsSL https://deno.land/x/install/install.sh | sh

The command above requires having both curl and unzip installed on our system.

After the command finishes running, we should see the following output:

Archive:  /home/<your_username>/.deno/bin/deno.zip
  inflating: /home/<your_username>/.deno/bin/deno
Deno was installed successfully to /home/<your_username>/.deno/bin/deno
Manually add the directory to your $HOME/.bash_profile (or similar)
  export DENO_INSTALL="/home/<your_username>/.deno"
  export PATH="$DENO_INSTALL/bin:$PATH"

As the message suggests, we need to edit (or create) the profile file used by our shell, for example $HOME/.profile or $HOME/.bash_profile, and add these two lines at the bottom:

export DENO_INSTALL="/home/<your_username>/.deno"
export PATH="$DENO_INSTALL/bin:$PATH"

Make sure to paste the lines actually displayed in your terminal by Deno.

After saving the profile file and restarting the terminal, we should be able to write deno and press enter to see the following output:

$ deno
Deno 1.11.5
exit using ctrl+d or close()
>

As suggested, we can exit from Deno by pressing Ctrl and D together.

Additionally, running deno run https://deno.land/std/examples/welcome.ts in the terminal should display the following message:

$ deno run https://deno.land/std/examples/welcome.ts
Welcome to Deno!

If setting the PATH environment variable is not possible, we can still call deno by its relative path, for example:

$ .deno/bin/deno
Deno 1.11.5
exit using ctrl+d or close()
>

Now that Deno is correctly installed, we can start writing our script.

Step 1: Create the main.ts file

In an empty directory, let's create a file named main.ts (the name main has no importance) that looks like this:

// main.ts

console.log("Hello from Deno!");

Let's open a terminal in the same directory and run deno run main.ts to see the following output:

$ deno run main.ts
Check file:///home/<your_username>/<some_directory>/main.ts
Hello from Deno!

As we can see, we don't need a package.json file or any external dependencies to run this TypeScript file.

We can also run deno fmt main.ts or use the Deno extension for VSCode to automatically format the source code in this file.

Step 2: Define the base URL for the Hacker News API

Since we will interact with the Hacker News API, let's clear the main.ts file and define the baseURL variable as follows:

// main.ts

/** Base URL for all calls to the Hacker News API */
const baseURL = "https://hacker-news.firebaseio.com/v0";

This URL is the target for the HTTP requests we will make in later steps.

Step 3: Define the Item interface

The Hacker News API represents user-generated content, including comments, as items with various properties. We can identify items using the properties id, a unique incrementing integer number, and type, an enumeration of different item types.

Following the official API specification, let's model an item using a TypeScript interface:

// main.ts

const baseURL = "https://hacker-news.firebaseio.com/v0";

/** Item represents an item fetched from the HN API */
interface Item {
  /** Unique item ID; the only required property */
  readonly id: number;
  /** Item type (`job`, `story`, `comment`, `poll`, or `pollopt`) */
  readonly type?: string;
  /** Username of the user who submitted the item */
  readonly by?: string;
  /** Title text for a story, poll or job */
  readonly title?: string;
  /** URL for a story or job */
  readonly url?: string;
  /** Text for a story, comment, poll, poll option or job */
  readonly text?: string;
  /** Unix timestamp for when the item was created */
  readonly time?: number;
  /** Score for a story, job or poll; votes for a poll option */
  readonly score?: number;
  /** Number of total comments for a story or poll */
  readonly descendants?: number;
  /** Set to `true` if the item is deleted */
  readonly deleted?: boolean;
  /** Set to `true` if the item is dead */
  readonly dead?: boolean;
  /** ID of the parent item of a comment (a story or another comment) */
  readonly parent?: number;
  /** List of IDs of the item's comments, in display order */
  readonly kids?: number[];
  /** ID of the poll associated to a poll option */
  readonly poll?: number;
  /** List of IDs of related poll options, in display order */
  readonly parts?: number[];
}

Note that only the id property is required, all other properties are marked as optional with ?. All properties are also marked as readonly so that they cannot be reassigned.

We won't be using all the properties present in Item, but they have been defined and documented for completeness.

Step 4: Get the ID of the most recent item

Since item IDs are represented as an incrementing integer, the most recent item is the one with the largest ID. To get this value, we can use the /maxitem.json endpoint, that is https://hacker-news.firebaseio.com/v0/maxitem.json.

Let's implement an asynchronous function named getMaxItemID() that fetches this data:

// main.ts

const baseURL = "https://hacker-news.firebaseio.com/v0";

interface Item {
  // Omitted for brevity
}

/** getMaxItemID returns the ID of the most recent item published on HN */
async function getMaxItemID(): Promise<number> {
  const endpoint = `${baseURL}/maxitem.json`;
  const res = await fetch(endpoint);
  const id = await res.json();
  return id;
}

We first define the endpoint, then use the fetch web API implemented by Deno to retrieve the JSON data representing the largest item ID.

As a quick test, let's also add this line below our function:

// main.ts

// ...

async function getMaxItemID(): Promise<number> {
  const endpoint = `${baseURL}/maxitem.json`;
  const res = await fetch(endpoint);
  const id = await res.json();
  return id;
}

// Remove this line after this step
console.log(await getMaxItemID());

Now let's run deno run main.ts:

$ deno run main.ts
error: Uncaught (in promise) PermissionDenied: Requires net access to "hacker-news.firebaseio.com", run again with the --allow-net flag
  const res = await fetch(endpoint);
                    ^
    at deno:core/core.js:86:46
    at unwrapOpResult (deno:core/core.js:106:13)
    at Object.opSync (deno:core/core.js:120:12)
    at opFetch (deno:extensions/fetch/26_fetch.js:43:17)
    at mainFetch (deno:extensions/fetch/26_fetch.js:170:61)
    at deno:extensions/fetch/26_fetch.js:395:7
    at new Promise (<anonymous>)
    at fetch (deno:extensions/fetch/26_fetch.js:357:15)
    at getMaxItemID (file:///home/<your_username>/<some_directory>/main.ts:43:21)
    at file:///home/<your_username>/<some_directory>/main.ts:48:1

Since Deno is secure by default, it prevented our script from accessing the network connection to fetch data from hacker-news.firebaseio.com. As explained in the error message, we need to use the --allow-net flag to explicitly grant this permission when running the script:

$ deno run --allow-net main.ts
27121843

Now that the script can reach the API, we can see the ID of the most recent item being printed in the console.

Before going to the next step, let's remove the line console.log(await getMaxItemID()); that we just added.

Step 5: Get the most recent item by its ID

Now that we have the ID of the most recent item, we can get the item itself by using the /item/<id>.json endpoint, for example https://hacker-news.firebaseio.com/v0/item/27121843.json.

Let's implement another asynchronous function named getItemByID() that fetches an item given its ID:

// main.ts

// ...

/** getItemByID fetches an item from the HN API given its ID */
async function getItemByID(id: number): Promise<Item | undefined> {
  const endpoint = `${baseURL}/item/${id}.json`;
  const res = await fetch(endpoint);
  const item = await res.json();
  return item ?? undefined;
}

If the ID is valid, getItemByID() will return an object described by the Item interface; otherwise, it will return undefined. The nullish coalescing operator (??) is supported out of the box by Deno.

We can now fetch and print the latest item by adding the line console.log(await getItemByID(await getMaxItemID())); as shown:

// main.ts

// ...

async function getItemByID(id: number): Promise<Item | undefined> {
  const endpoint = `${baseURL}/item/${id}.json`;
  const res = await fetch(endpoint);
  const item = await res.json();
  return item ?? undefined;
}

// Remove this line after this step
console.log(await getItemByID(await getMaxItemID()));

Running again our script with deno run --allow-net main.ts should display an item similar to this:

$ deno run --allow-net main.ts
Check file:///home/<your_username>/<some_directory>/main.ts
{
  by: "velut",
  id: 27121843,
  parent: 27121265,
  text: "Thank you!<p>Yes, I use prism-react-renderer for syntax highlighting.<p>I did not know about Shiki, ...",
  time: 1620758446,
  type: "comment"
}

Note that the item displayed in your terminal will be different and won't necessarily be a comment. Sometimes it may also be undefined if the HN API has not properly updated yet internally.

Step 6: Stream comments from Hacker News

Now that we have the necessary building blocks, let's implement the streamComments() function.

First, let's get the ID of the most recent item:

// main.ts

// ...

/** streamComments continuously fetches and displays the most recent comments published on HN */
async function streamComments() {
  // Get the ID of the most recent item
  let id = await getMaxItemID();
}

Then, get the item, increment the ID for the next item and loop forever:

// main.ts

// ...

async function streamComments() {
  let id = await getMaxItemID();

  // Keep running forever
  while (true) {
    // Get the item
    const item = await getItemByID(id);

    // Increment the ID for the next item
    id += 1;
  }
}

However, we need to wait a little if an item still does not exist. To do so, let's import a remote module containing a sleep function:

// main.ts

// Import the `sleep` function from a remote module
import { sleep } from "https://deno.land/x/sleep/mod.ts";

// ...

async function streamComments() {
  let id = await getMaxItemID();

  while (true) {
    const item = await getItemByID(id);

    // If the item does not exist, wait 5 seconds and try again
    if (!item) {
      await sleep(5);
      continue;
    }

    id += 1;
  }
}

To import modules in our Deno scripts, we use file paths for local modules (for example, import ... from "./some/local/module.ts") and URLs for remote modules (for example, import ... from "https://example.com/some/remote/module.ts).

We don't need to install packages from a centralized remote repository like npm, Deno will instead download and cache the remote modules and their dependencies the first time they are used.

Let's now complete the streamComments() function by printing only comments:

// main.ts

import { sleep } from "https://deno.land/x/sleep/mod.ts";

// ...

async function streamComments() {
  let id = await getMaxItemID();

  while (true) {
    const item = await getItemByID(id);
    if (!item) {
      await sleep(5);
      continue;
    }

    // Print only items that are visible comments
    const { type, deleted, dead } = item;
    const removed = deleted || dead || false;
    if (type === "comment" && !removed) {
      // Get the comment's author, if any
      const author = item.by ?? "unknown author";

      // Get the comment's URL on HN website
      const hnURL = `https://news.ycombinator.com/item?id=${id}`;

      // Print the comment
      console.log(`\nRead new comment by ${author} on ${hnURL}`);
      console.log(item);
    }

    id += 1;
  }
}

Let's finish our script by calling await streamComments() at the bottom:

// main.ts

// ...

async function streamComments() {
  // ...
}

await streamComments();

You can compare your script with the final version of the script.

Step 7: Running the script

Now that our script is complete, we can run it with the same deno run --allow-net main.ts command used before:

$ deno run --allow-net main.ts
Download https://deno.land/x/sleep/mod.ts
Warning Implicitly using latest version (v1.2.0) for https://deno.land/x/sleep/mod.ts
Download https://deno.land/x/sleep@v1.2.0/mod.ts
Download https://deno.land/x/sleep@v1.2.0/sleep.ts
Download https://deno.land/std/log/mod.ts
<...many other downloads...>
Check file:///home/<your_username>/<some_directory>/main.ts

Read new comment by tonyedgecombe on https://news.ycombinator.com/item?id=27762774
{
  by: "tonyedgecombe",
  id: 27762774,
  parent: 27761864,
  text: "&gt;The printer stack which has barely received an update since Windows XP is a perfect example.<p>T...",
  time: 1625674491,
  type: "comment"
}

Read new comment by jph on https://news.ycombinator.com/item?id=27762775
{
  by: "jph",
  id: 27762775,
  parent: 27762699,
  text: "The catchphrase is &quot;neurons that fire together wire together&quot;.",
  time: 1625674509,
  type: "comment"
}

Read new comment by exo-pla-net on https://news.ycombinator.com/item?id=27762777
{
  by: "exo-pla-net",
  id: 27762777,
  parent: 27762326,
  text: "The plant kingdom is almost entirely mono-food-sourced. They almost all rely on the sun, which, pend...",
  time: 1625674519,
  type: "comment"
}

As we can see, Deno first downloads the required remote modules and then starts running our script. After a few seconds, we should be able to read the stream of comments being published to Hacker News in real time.

Conclusion

In this post, we discovered Deno, a simple, modern and secure alternative to Node.js, and explored many of its built-in features such as TypeScript support, web APIs, ES modules imports and modern Javascript syntax. These features allowed us to quickly implement and run a small script that fetches data in real time from a third party API.

You can learn more about Deno from its website and manual. When developing more complex scripts, you may also want to use Deno's standard library or search for third party modules.

If you liked this article and want to know when I post more, you can follow me on Twitter.

Credits

• Cover photo by Katie Smith on Unsplash

What is npm?

The term npm refers both to:

• The npm CLI tool installed by default with Node.js on your machine
• The npm registry, an online service which collects more than 1.6M public Javascript packages

For example, when running the npm install react command in your Javascript project, you are downloading the react package from the online npm registry.

In this post, we are interested in the online service and its public API.

What is the npm registry API?

While many people regularly use npm's website to discover packages, only a few know that npm also provides a public REST API accessible at registry.npmjs.org.

This API provides methods to:

• Get information about the registry itself
• Get all available information about a specific package
• Get information about a specific version of a package
• Search packages by text
• Count the number of downloads for packages

We can call these methods by:

• Sending HTTP requests to the correct endpoints
• Using a fully typed package I developed named query-registry

Before we start

You can follow along online on this RunKit notebook or on your machine by installing the isomorphic-unfetch and query-registry packages as follows:

npm install isomorphic-unfetch query-registry

You can also use as references the official API specification and the documentation for query-registry.

Finally, you can explore the API and its responses in your browser by going to registry.npmjs.org.

Example 1: Get information about the registry itself

Endpoint

If we want to know more about the underlying database used by the registry, we can send a GET request to the / endpoint, that is https://registry.npmjs.org/.

With fetch

async function example1WithFetch() {
  const endpoint = "https://registry.npmjs.org/";
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function example1WithQueryRegistry() {
  const data = await queryRegistry.getRegistryMetadata();
  console.log(data);
}

Response

We receive a response containing information about the registry's database, including its name and some interesting attributes, as shown below:

{
  "db_name":"registry",
  "engine":"couch_bt_engine",
  "doc_count":2226548,
  "doc_del_count":334,
  "update_seq":5769731,
  "purge_seq":0,
  "compact_running":false,
  "sizes":{
    "active":57693928578,
    "external":132154863659,
    "file":58937123056
  },
  "disk_size":58937123056,
  "data_size":57693928578,
  "other":{
    "data_size":132154863659
  },
  "instance_start_time":"1624686290809498",
  "disk_format_version":7,
  "committed_update_seq":5769731,
  "compacted_seq":5729968,
  "uuid":"964c127ddcbbd59982db296a0f9e8a56"
}

Example 2: Get all available package metadata

Endpoint

If we want to get a packument (package document) containing all the information available on a package, we can send a GET request to the /<package> endpoint, for example https://registry.npmjs.org/react or https://registry.npmjs.org/@types/node.

With fetch

async function example2WithFetch(name) {
  const endpoint = `https://registry.npmjs.org/${name}`;
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function example2WithQueryRegistry(name) {
  const data = await queryRegistry.getPackument({ name });
  console.log(data);
}

Response

We receive a response containing all the data associated to a package, including its ID, name, description, author, license and manifests for each published version.

{
  "_id": "react",
  "_rev": "1684-29eba7dd741dee3c165b86b7e4f63461",
  "name": "react",
  "description": "React is a JavaScript library for building user interfaces.",
  "dist-tags": {…},
  "versions": {…},
  "maintainers": […],
  "time": {…},
  "repository": {…},
  "readme": "",
  "readmeFilename": "",
  "homepage": "https://reactjs.org/",
  "keywords": […],
  "bugs": {…},
  "users": {…},
  "license": "MIT"
}

Example 3: Get information about a specific version of a package

Endpoint

If we want to get a package manifest containing information about a specific version of a package, for example react@17.0.2 or @types/node@15.14.0, we can send a GET request to the /<package>/<version> endpoint, for example https://registry.npmjs.org/react/17.0.2 or https://registry.npmjs.org/@types/node/15.14.0.

With fetch

async function example3WithFetch(name, version) {
  const endpoint = `https://registry.npmjs.org/${name}/${version}`;
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function example3WithQueryRegistry(name, version) {
  const data = await queryRegistry.getPackageManifest({ name, version });
  console.log(data);
}

Response

We receive a response containing data that describes a published version of a package. This data consists of the contents of package.json at publishing time plus some additional attributes added by the registry.

{
  "name": "react",
  "description": "React is a JavaScript library for building user interfaces.",
  "keywords": […],
  "version": "17.0.2",
  "homepage": "https://reactjs.org/",
  "bugs": {…},
  "license": "MIT",
  "main": "index.js",
  "repository": {…},
  "engines": {…},
  "dependencies": {…},
  "browserify": {…},
  "_id": "react@17.0.2",
  "_nodeVersion": "15.11.0",
  "_npmVersion": "7.6.0",
  "dist": {…},
  "_npmUser": {…},
  "directories": {},
  "maintainers": […],
  "_npmOperationalInternal": {…},
  "_hasShrinkwrap": false,
}

Example 4: Search packages by text

Endpoint

If we want to search packages by text, we can send a GET request to the /-/v1/search?text=<some query> endpoint, for example https://registry.npmjs.org/-/v1/search?text=react.

We can also use special keyword parameters in our text query to improve our results. For example, to find packages that I published we can use the author:velut keyword parameter like this: https://registry.npmjs.org/-/v1/search?text=author:velut.

The official API specification contains the full list of supported search criteria.

With fetch

async function example4WithFetch(text) {
  const endpoint = `https://registry.npmjs.org/-/v1/search?text=${text}`;
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function example4WithQueryRegistry(text) {
  const data = await queryRegistry.searchPackages({ query: { text } });
  console.log(data);
}

Response

We receive a response containing a list of packages matching our query inside the objects attribute. Each package comes with a small number of important attributes, including name and version, plus some score values for the package itself and for its relevance to our query.

{
  "objects": [
    {
      "package": {
        "name": "react",
        "scope": "unscoped",
        "version": "17.0.2",
        "description": "React is a JavaScript library for building user interfaces.",
        "keywords": ["react"],
        "date": "2021-03-22T21:56:19.536Z",
        "links": {
          "npm": "https://www.npmjs.com/package/react",
          "homepage": "https://reactjs.org/",
          "repository": "https://github.com/facebook/react",
          "bugs": "https://github.com/facebook/react/issues"
        },
        "publisher": {
          "username": "…",
          "email": "…"
        },
        "maintainers": [
          { "username": "…", "email": "…" },
          { "username": "…", "email": "…" }
        ]
      },
      "score": {
        "final": 0.5866665170132767,
        "detail": {
          "quality": 0.5246016720020373,
          "popularity": 0.8931981392742823,
          "maintenance": 0.3333333333333333
        }
      },
      "searchScore": 100000.63
    }
  ],
  "total": 164637,
  "time": "Fri Jul 02 2021 13:13:14 GMT+0000 (Coordinated Universal Time)"
}

Example 5: Count downloads for a package

Endpoint

If we want to count the number of downloads for a package in a given time period, we can send a GET request to a slightly different API endpoint at https://api.npmjs.org/downloads/point/<period>/<package>, for example https://api.npmjs.org/downloads/point/last-week/react. Supported time periods include last-day, last-week, last-month and last-year.

The download counts API also provides other methods to count downloads for packages and for the whole registry.

With fetch

async function example5WithFetch(name, period) {
  const endpoint = `https://api.npmjs.org/downloads/point/${period}/${name}`;
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function example5WithQueryRegistry(name, period) {
  const data = await queryRegistry.getPackageDownloads({ name, period });
  console.log(data);
}

Response

We receive a simple response containing the package's name, its total number of downloads and the start and end dates for the selected time period.

{
  "downloads": 10889040,
  "start": "2021-06-25",
  "end": "2021-07-01",
  "package": "react"
}

Bonus: Using a registry mirror

Why use a mirror?

Sometimes we may want to use a proxy or mirror of the npm registry instead of the original registry itself. For example, Cloudflare provides a mirror at https://registry.npmjs.cf with CORS enabled, allowing us to query the registry directly from the browser or client-side applications.

For example, try pasting this snippet in your browser's console:

fetch("https://registry.npmjs.org/react").then(res => res.json()).then(console.log)

It should fail with a CORS error because it's using the original registry. Instead, the following snippet should work because it's using Cloudflare's registry mirror.

fetch("https://registry.npmjs.cf/react").then(res => res.json()).then(console.log)

Endpoint

We can use the same endpoints available on registry.npmjs.org provided that they are supported by the chosen mirror registry.

With fetch

async function bonusWithFetch(name) {
  const endpoint = `https://registry.npmjs.cf/${name}`;
  const res = await fetch(endpoint);
  const data = await res.json();
  console.log(data);
}

With query-registry

async function bonusWithQueryRegistry(name, registry) {
  const data = await queryRegistry.getPackument({ name, registry });
  console.log(data);
}

Response

The responses should be the same as the ones provided by the original npm registry, maybe slightly delayed due to the mirroring process.

Conclusion

In this post, we have learnt what is npm, how we can use its public API to discover and analyze public Javascript packages and how we can take advantage of API client wrappers such as query-registry and registry mirrors such as https://registry.npmjs.cf to improve our interactions with this API both in server-side and client-side Javascript applications.

Be sure to leave a comment if you have any doubts or if you end up building something interesting using this lesser known but powerful API.

If you liked this article and want to know when I post more, you can follow me on Twitter.

Credits

• Cover photo by Ricardo Gomez Angel on Unsplash