ML: Chatbot RASA: Introduction
Introduction
The RASA conversational systems engine allows the creation of chatbots for various specific tasks using modern deep learning methods. RASA consists of two parts: NLU (Natural Language Understanding) and Core (dialogue logic).
In NLU, the input text from the user is classified into one of the predefined classes, known as intents. For example, phrases like "hello" or "good afternoon" can belong to the same intent called greet. Regardless of how a person greets, the RASA core then operates solely with the intent name greet. In addition to intent classification, NLU extracts predefined entities from the text, such as names, numbers, etc., which can be used in the dialogue.
The possible bot responses also
have formal names (starting with the prefix utter_).
They are called action.
Each of these responses is associated with natural language text from the bot.
The training dialogue examples (stories),
consist of sequences of user intents and bot actions.
RASA must "remember" these sequences and learn to accurately
predict responses in unfamiliar situations.
Scripts written in
Python can be used as responses.
They implement complex behavior or retrieve information from a database.
Simple example
| config.yml | domain.yml +-data | nlu.yml | rules.yml | stories.ymlAfter installing RASA, you should run "the rasa init" command in the console, which will prompt you to enter a directory name and, after answering "Y", will create a new project.
When asked: "Do you want to train an initial model?" respond with "N" (no). Edit the data/nlu.yml file in this project with the list of intents
(hereafter, the optional first line version: "2.0" is omitted):
nlu: # data/nlu.yml
- intent: greet # intent is named greet
examples: | # user 🙎 greets the bot
- hi # these are examples of a greeting
- hello
- good afternoon
- intent: goodbye # intent is named goodbye
examples: | # user 🙎 says goodbye to the bot
- bye # these are examples of a goodbye
- goodbye
- see you tomorrow
This bot will understand two intents from the user (greeting and saying goodbye).
In practice, the more examples provided for each intent, the better RASA
learns to classify them.
In the root file domain.yml, in the responses section,
list the bot's textual responses:
responses: # domain.yml
utter_greet: # response name
- text: Hello, nice to meet you. # one of these phrases will be randomly chosen
- text: I'm so glad to meet you!
utter_goodbye:
- text: See you soon!
image: "https://i.imgur.com/nGF1K8f.jpg"# send an image along with the text
All RASA version 2.0 files use yaml syntax. In YAML, indentation and horizontal alignment of blocks are crucial. Therefore, it's advisable not to use tabs (they should be replaced with spaces). After the '#' symbol, a line comment follows, which RASA ignores. In lists, there should be a space after the dash "-" and if the text contains a colon or dash (yaml file markup elements), it must be enclosed in double quotes "...". If something isn't working as expected, you can check the file with the yamlchecker.com validator.
Stories are examples of dialogues on which the bot learns to respond appropriately depending on the conversation's history. In the data/stories.yml file, let's create a single story for now (in reality, examples of many stories are described):
stories: # data/stories.yml - story: hello and goodbye # arbitrary description of the story's content steps: - intent: greet # 🙎 hello - action: utter_greet # 💻 Hello, nice to meet you - intent: goodbye # 🙎 see you tomorrow - action: utter_goodbye # 💻 See you soon!
Finally, in the data/rules.yml file, delete all lines under the "rules:" section for now, and change the language to English in the settings file config.yml:
language: en
After training RASA (by running the "rasa train" command in the console), you can chat with the bot (use the "rasa shell" command):
Your input -> helo Hello, nice to meet you Your input -> seeya See you soon!Note the typo in "helo" and the unfamiliar word "seeya", which the NLU engine handled quite well.
To interact with the bot, instead of "rasa shell", you can also use the !agent.py script. It allows you to track more information than just the dialogue with the bot and saves the interaction logs in the !agent.log file. This script, along with the examples in this section, is located in the project root Bot01_Simple.zip. If the information provided by the !agent.py script is insufficient, try the standard debug mode "rasa shell --debug" :).
Intent training
To classify a user's phrase into one of the intents,
a complex neural network is trained in the
NLU module.
For this to work, input texts are transformed into vectors (arrays)
of real numbers called futures.
When building the feature vector,
technologies such as bag of words,
word vectorization like embedding (word2vec),
and N-gram letters (individual lettersN=1, pairs of consecutive letters
N=2, etc.) are used.
The corresponding settings in the config.yml
file allow the use of pre-trained word vectors for the given language, enabling the
NLU to understand synonyms and semantically similar words
that are not explicitly mentioned in the intent examples.
Thanks to N-grams, RASA effectively handles typos in words.
The more examples provided for a given intent, the better (usually) the system learns. However, there's a risk of overlap between similar phrases across different intents, which can reduce classification quality. Therefore, the data/nlu.yml file with intents (as well as stories) should be tested from time to time. More details about this will be discussed at the end of this document.
During training (as indicated by the progress bar on the screen after running "rasa train") it's important to monitor training errors (t_loss, i_acc). If t_loss (total loss) is significantly greater than 1.5, and/or i_acc (intent accuracy) is significantly less than one, it may be worth increasing the number of training epochs in the pipeline (for example, doubling them):
pipeline: # config.yml
# ...
- name: DIETClassifier # intent and entity classifier
epochs: 200 # number of training epochs
If this doesn't help, you should analyze the examples in the intens to check for excessive similarity
and conduct an analysis of the testing results of the project.
Entities
Intents can contain entities, which the NLU engine can extract from the text. For example, let's add a third intent my_name to the data/nlu.yml file:
- intent: my_name # data/nlu.yml
examples: |
- my name is [Anna](PERSON)
- my name is [Anastasia](PERSON)
- call me [Bond](PERSON)
- call me [Izya](PERSON)
- my friends call me [Al](PERSON)
- my friends call me [Kitty](PERSON)
- [Marry](PERSON)
- [Alex](PERSON)
Inside square brackets is a part of the text (in the example, a specific name),
and immediately following in round brackets (without a space)(!!!) is the name of the entity.
There should be no spaces within the round brackets!
In the domain.yml file, you must list all intents
and the entities used within them:
intents: # domain.yml - greet # listed intents - goodbye - my_name entities: - PERSON # entities extracted from intentsAfter training (rasa train), you can test entity extraction in a non-dialog mode (simply by entering phrases) using the script !nlu_debug.py:
My name is Anny my_name [1.000], greet [0.000], goodbye [0.000], PERSON=Anny [1.00], DIETClassifierThe second line shows a list of intents with their classification confidence levels, followed by the extracted entities (if any) with their confidence levels and the name of the classifier that performed the extraction.
Batch testing is possible with the script !nlu_debug_file.py, which takes texts from !nlu_debug_file.txt. The script!agent.py provides similar information during a dialogue. All these scripts are available in the project Bot02_PERSON.zip. The simplest way to test entities using RASA is to use slots.
Slots
Slots are the "memory cells" of a bot, which can be of various types. They can store entity values or any other information. For example, let's add a slots section to the domain.yml file:
slots: # domain.yml
PERSON: # slot name (can be in any language)
type: text # slot type (this is a text string)
Here, a variable (slot) named PERSON is defined.
This name matches the name of the PERSON entity introduced earlier, so when the
PERSON entity is extracted from the text, its value will be stored in the
PERSON slot.
If the slot name does not match
the entity name, it can be filled in a form
or a Python action.
Slots can be used in the bot's responses by enclosing their name in curly brackets:
responses: # domain.yml
utter_pleased_to_meet_you:
- text: Pleased to meet you, {PERSON}!
To make this response work, it needs to be added to the story:
stories: # data/stories.yml - story: greet, name, and goodbye steps: - intent: greet # 🙎 hello - action: utter_greet # 💻 Hello, nice to meet you - action: utter_what_is_your_name # 💻 What is your name? - intent: my_name # 🙎 my name is Ann (from nlu.yml) - action: utter_pleased_to_meet_you # 💻 Pleased to meet you, Ann (from domain.yml) - intent: goodbye # 🙎 see you tomorrow - action: utter_goodbye # 💻 See you soon!After training (rasa train), you can chat with this bot using rasa shell or !agent.py. Slots are described in more detail in the document "Slots and forms", and their usage in stories is covered in the document "Stories and rules"
Entity parameters
When describing an entity, you can use extended syntax in curly (rather than parentheses) brackets:
- my conversation rating is [positive]{"entity":"RATING","value":"positive"}
where the following properties are available:
[<entity-text>]{"entity": "<ENTITY_NAME>",
"value": "<ENTITY_VALUE>",
"role": "<ENTITY_ROLE>",
"group": "<ENTITY_GROUP>" }
✒ The value property is convenient for identifying identical entity values written in different ways. If only the value property is needed, you can still use parentheses in the format [text](ENTITY_NAME:value) (there should be no spaces inside the parentheses). For example, in some languages, such as Russian, nouns have both gender and case. Let's explore this:
- intent: number
examples: |
- [один]{"entity":"NUMBER", "value": "1"}
- [одну](NUMBER:1)
- [1](NUMBER) # value will be taken from [1]
- [2](NUMBER)
- [два](NUMBER:2)
- [две](NUMBER:2)
As a result, in the phrase "Хочу две пиццы" the NUMBER entity with
value=2 will be extracted, which can then be used based on its value, without worrying about the word form "две".
It's advisable to add such phrases with markup to the corresponding intent as well:
- intent: i_want_to_buy
examples: |
- Хочу [две](NUMBER:1) [пиццы](ITEM:pizza)
- Буду [пять](NUMBER:5) [пицц](ITEM:pizza)
- [153]{NUMBER:153} [пиццы](ITEM:pizza)
RASA will learn to extract the NUMBER entity
across both intents. In the case of typos or unaccounted word forms, the extracted (correct) entity might not have the
value property.
✒ The "role" property allows specifying the role an entity plays in the text. For example, consider an intent with the following example (it should be in one line):
- I want to fly from [Dnipro]{"entity":"CITY", "role": "from"}
to [Paris]{"entity":"CITY", "role": "to"}.
In a similar phrase, NLU RASA will return two entities named CITY.
The first one with value=Dnipro and role=from, and the second with
value=Paris
and role=to.
✒ The group property allows specifying the group number of an entity chain (this should also be in one line):
- I want a [small]{"entity": "SIZE", "group": "1"}
[pizza]{ "entity": "ITEM", "value": "pizza"}
with [mushrooms]{ "entity": "TOPPING", "group": "1"} and another
[large]{ "entity": "SIZE", "group": "2"}
with [cheese]{ "entity": "TOPPING", "group": "2"}.
- And in the second one, add [tomatoes]{"entity": "TOPPING", "group": "2"}.
When using roles and groups, they need to be described when declaring the entity:
entities:
- CITY:
roles:
- from
- to
- TOPPING:
groups:
- 1
- 2
Synonyms and lookup tables
There is another mechanism, different from the value property, for aligning entities with different forms of expression. For example, if we want the entity NUMBER to consistently return NUMBER=два in texts regardless of the grammatical case (два, две, двух, двумя in Russian), we can write the following in data/nlu.yml:
- synonym: два # data/nlu.yml
examples: |
- две
- двух
- двумя
For synonyms to work, the EntitySynonymMapper must be included in the pipeline section of the config.yml file.
☝ Synonym mapping occurs after entity extraction. This means that training examples must include the synonyms so that the model can learn to recognize them as entities and then replace them with the same value. Therefore, the synonym mechanism is useful to avoid specifying the value property in every training example.
Another way to help NLU RASA extract entities is by creating lookup tables (also in data/nlu.yml).
- lookup: PERSON # data/nlu.yml
examples: |
- alex
- tom
- michael
- nick
In this example, the PERSON entity will reliably extract
the names listed.
Lookup tables are convenient when the list of entity examples is extensive.
In this case, they serve as a "database."
However, lookup tables search for the exact text specified in the examples.
Unlike examples in intent training,
entities from lookup tables will not be extracted if there are typos.
To enable lookup tables, the following must be added to the pipeline:
pipeline: # config.yml
#...
- name: RegexFeaturizer
case_sensitive: false
use_lookup_tables: true # enables lookup tables
use_regexes: true
Rules
Rules contain templates for short dialogue segments that must always follow the same path. A rule can include only one user intent, followed by one or more bot responses (action). It's important not to overuse rules because they represent the "non-trainable" part of stories (RASA strictly follows them when they are triggered). To enable rules, the following must be added to config.yml:
policies: # config.yml - name: RulePolicyIf the policies section is entirely empty, this step can be skipped since the default policies will be used, which include RulePolicy. Rules are listed in the data/rules.yml file. For example:
rules: # data/rules.yml - rule: respond to greeting # description of the rule's content steps: - intent: greet # 🙎 hi - action: utter_greet # 💻 Hello, nice to meet you!Now, the greeting doesn't need to be included in the story examples, as it is handled by a "reflexive response" in the form of a rule. Rules are discussed in more detail in the document "Stories and rules".
Regular expressions
Regular expressions provide additional features for classifying intents and extracting entities from them. The first task utilizes the RegexFeaturizer, which must be added to the pipeline in the config.yml file, while the second task uses the RegexEntityExtractor (also in the same file).
When you need to emphasize the presence of specific text in intent examples, a regular expression with an arbitrary name is created:
nlu:
- regex: help_important_for_intend # data/nlu.yml
examples: |
- \bhelp\b # the presence of the word 'help' is important
The name of the regular expression for entity extraction by RegexEntityExtractor must match the entity name:
nlu: # data/nlu.yml
- regex: ACCOUNT_NUMBER
examples: |
- \d{10,12} # from 10 to 12 digits
- intent: inform_account_number
examples: |
- My account number is [1234567891](ACCOUNT_NUMBER)
- This is my account number: [1234567891](ACCOUNT_NUMBER)
It’s worth noting that regular expressions are not always convenient for extracting numbers. They help extract the ACCOUNT_NUMBER entity from any intent without necessarily identifying the specific intent: inform_account_number. Additionally, this may lead to the duplication of the ACCOUNT_NUMBER entity by two classifiers: RegexEntityExtractor and DIETClassifier. In fact, the DIETClassifier learns to identify any integers from the given examples, so it is often possible to do without the RegexEntityExtractor.
Multiple intents
Often, people in chat write several sentences, each of which can be a standalone intent (Multi-Intent Classification). To handle these in the pipeline, the tokenizer needs to include the following properties:
pipeline: # config.yml
- name: WhitespaceTokenizer
intent_tokenization_flag: true
intent_split_symbol: "+"
Next, create a "combined" intent, where the name consists of several existing intents connected by a plus sign.
You don't need to describe many examples in it, as they will be pulled from the respective intents:
nlu: # data/nlu.yml
- intent: greet+my_name
examples: |
- hi. my name is [Anny](PERSON)
- hello. my name is [Anna](PERSON)
The intent greet+my_name needs to be added to the intents section of the
domain.yml file and then used in stories or rules as usual:
stories: # data/stories.yml
- story: hello, my name is
steps:
- intent: greet+my_name # Hello, my name is Anastasia
- slot_was_set:
- PERSON
- action: utter_greet # Hello.
- action: utter_glad_to_meet_you # Nice to meet you, Anastasia
Button-based dialogues
Sometimes, you expect specific responses from a user. In such cases, making them type out a response may not always be practical, so you can use a menu in the form of buttons, which is common in simpler bots.
For example, let's ask the client to rate the quality of the conversation at the end. To do this, we add the following to the responses:
responses: # domain.yml
utter_how_is_our_conversation:
- text: "Rate the quality of our conversation:" # text preceding the buttons
buttons:
- title: Excellent # text on the first button
payload: /perfect # /<intent name> client intent
- title: Not so good # text on the second button
payload: /awful
utter_happy_for_us:
- text: I'm so happy
utter_so_sorry:
- text: I'm very sorry
In the payload field, the intent name follows the slash "/" and is what the button menu triggers
(this intent, as usual, should be added to domain.yml under the intents section).
Now, in the stories, you can write something like:
stories: # data/stories.yml - story: Hello and goodbye, excellent steps: - intent: greet # 🙎 Hello - action: utter_greet # 💻 Hello, nice to meet you - intent: goodbye # 🙎 See you tomorrow - action: utter_goodbye # 💻 See you soon! - action: utter_how_is_our_conversation # Rate the quality of our conversation - intent: perfect # 🙎 Button <Excellent> - action: utter_happy_for_us # 💻 I'm so happy - story: Hello and goodbye, awful steps: - intent: greet # 🙎 Hello - action: utter_greet # 💻 Hello, nice to meet you - intent: goodbye # 🙎 See you tomorrow - action: utter_goodbye # 💻 See you soon! - action: utter_how_is_our_conversation # Rate the quality of our conversation - intent: awful # 🙎 Button <Not so good> - action: utter_so_sorry # 💻 I'm very sorry
A button can also send both intent and entities:
buttons:
- title: Excellent
payload: /perfect{{"SCORE":"5","INFO":"Doing quite well"}}
Note the use of double curly braces. This is a mechanism for escaping in
JSON.If the button menu is generated within an action as a string like "/perfect{\"SCORE\":\"5\"}", then single braces should be used.
Of course, buttons may not look great in the console. However, in production, depending on the environment, they will look fine. For example, a Telegram bot will display the familiar button layout, and you can even set the button orientation. A complete project that uses a combination of text input and button menus can be found in the project Bot03_Buttons.zip. Please note that to select a button, you should use the up and down arrow keys, rather than trying to enter the button number. A more complex project with multiple button menus (including dynamic ones) can be downloaded from Bot_Pizza_Buttons.zip. For testing, !agent.py is not suitable, so you should use rasa shell.
Tests
To test the performance of your bot, it is not necessary to manually input text every time (in the "rasa shell" mode or !agent.py). Instead, you can create a set of test stories in the file tests/test_stories.yml, which can be checked in batch mode using the "rasa test" command. For example, for the stories mentioned at the beginning of this document, the tests might look like this:
stories: # tests/test_stories.yml
- story: 1. hello and goodbye
steps:
- user: | # what the user says
hello
intent: greet # the intent that should be recognized
- action: utter_greet # the expected bot response
- user: |
goodbye
intent: goodbye
- action: utter_goodbye
#----------------------------------------------------------
- story: 2. hello and goodbye # this is an incorrect test!!!
steps:
- user: |
goodbye!
intent: goodbye
- action: utter_greet
The results of the "rasa test" command can be found in files within the results folder.
Specifically, stories that fail the test are listed in the failed_test_stories.yml file:
stories: # results/failed_test_stories.yml - story: 2. hello and goodbye (.\tests\test_stories.yml) steps: - intent: goodbye - action: utter_greet # predicted: utter_goodbyeIn our example, the second test failed (as noted in the comment) because we trained RASA to respond with utter_goodbye to the goodbye intent, not with utter_greet (as specified in the test).
When writing stories, it is important to specify the extracted entities (if applicable) with the correct values, roles, and groups (if these were trained in RASA):
- user: |
[five](NUMBER:5) bottles of [Fanta](ITEM:Fanta)
intent: number_of_items
When slots are changed in actions (usually in custom Python actions),
this should also be indicated using slot_was_set (for more details, see "Stories and rules").
If the file results/failed_test_stories.yml is not empty after running "rasa test" and there are no comments about prediction errors, it is possible that incorrect entity properties were specified. For "debugging," you can start by shortening the story until it passes, and then identify the problem in the removed section.
In addition to the failed_test_stories.yml file, it is also worth reviewing other files, especially the illustrative PNG images. These images are grouped into three categories: intents (intent_...), entities (DIETClassifier_...) and stories (story_...). Each group includes two types of images:
- Confusion matrix (..._confusion_matrix.png):
This matrix has classes (e.g., intents) along the axes.
The intersection of row R and column C shows the number of times class R was classified as class C.
A well-trained model should have a diagonal confusion matrix (R==C). - Confidence Histogram (..._histogram.png):
This histogram shows the distribution of confidence scores (along the Y axis) of predictions.
Correct predictions are represented by blue bars, while incorrect ones are shown in red.
A good model has many blue bars (correct predictions) that are positioned high (indicating confident predictions).
Some useful commands for training and testing:
- rasa train nlu - trains only the intents (nlu.yml)
- rasa train core - trains only the core (stories.yml)
- rasa test nlu - tests only the intents (nlu.yml)
- rasa test core - tests only the core (stories.yml)
- rasa data validate - checks for errors or inconsistencies in all input data (domain, nlu, stories)
Some problems and their solutions
- "RuntimeError: coroutine raised StopIteration" -
this error occurs after starting core training.
It may be due to a slot in the stories or rules that is not defined in the slots section of the domain.yml file. - "TypeError: 'NoneType' object is not iterable" - this error occurs at the beginning of training.
It may happen if there is a - slot_was_set: section in the stories without a list. - "TypeError: '>' not supported between instances of 'int' and 'str'"
- this error occurs during core training.
Try clearing the policies in the config.yml file. - "TypeError: unsupported operand type(s) for +=: 'NoneType' and 'list' - this error occurs at the beginning of training.
It may happen if you forgot to define a custom action in the actions section. - "UserWarning: I The item '<..>' contains an example that doesn't start with a '-' symbol:" - this warning occurs when a line with spaces (instead of being completely empty) follows the list of intent examples in the nlu file.
- "UserWarning: Slot <..> was referenced by its name only. As <..> is of type categorical..." -
in stories, a categorical slot STATE is referenced in the slot_was_set section without a value. You need to either specify an explicit value or omit it if it is null.
What's next
The following documents discuss various aspects of the RASA framework in more detail:
- "Stories and rules" - describes methods for constructing stories and rules.
- "Slots and forms" - describes the bot’s memory cells (slots) and their combination into forms (forms).
- "Actions" - these are complex bot responses that are formatted as Python code.
- "NLU pipeline" - a look under the "hood" of RASA.
The following external links may also be useful:
- "Fallback and Human Handoff" - how to handle various bot failures;
- Getting Back on the Happy Path;
- How to Build a “Human” Handoff Feature;
- Source code of NLU components;
- A good description of regular expressions.
