Tokenizers

Start by creating a new project with npm init. Name your bot "weatherbot" (or something similar), and install the Slack and Natural APIs with the following command:

npm install @slack/client natural –save

Copy our Bot class from the previous chapters and enter the following in index.js:

'use strict';

// import the natural library
const natural = require('natural');

const Bot = require('./Bot');

// initalize the tokenizer
const tokenizer = new natural.WordTokenizer();

const bot = new Bot({
  token: process.env.SLACK_TOKEN,
  autoReconnect: true,
  autoMark: true
});

// respond to any message that comes through
bot.respondTo('', (message, channel, user) => {

  let tokenizedMessage = tokenizer.tokenize(message.text);

  bot.send(`Tokenized message: ${JSON.stringify(tokenizedMessage)}`, channel);
});

Start up your Node process and type a test phrase into Slack:

The returned tokenized message

Through the use of tokenization, the bot has split the given phrase into short fragments...