Chatbot (NLP)
Intent-based bot trained from JSON corpus.
- ROLE
- Developer
- DOMAIN
- Student work
OVERVIEW
An intent-based chatbot trained from a JSON corpus of intents, each with example patterns and responses. Input is tokenized, stemmed, and turned into a bag-of-words vector; a small feedforward network picks the intent, and a low-confidence prediction gets a fallback reply instead of a guess. Adding an intent means editing the JSON and retraining.
ARRIVED AS
“Lightweight Q/A without heavy infra.”
Before a chatbot can answer, it has to work out what the user wants: their intent. I wanted a small, accurate intent classifier that didn't need heavy infrastructure or a cloud NLP service.
Rule-based bots match keywords, and that breaks as soon as people phrase things differently. Machine learning picks up patterns instead, so phrases that mean the same thing land on the same intent. This bot uses classical NLP and a neural network, trained on a custom JSON corpus that maps example phrases to intents.
I wanted a fast prototype I could deploy for FAQs, customer service automation and small utilities, with low latency and few resources.
WHAT I BUILT
- 01Tokenization, stemming, bag-of-words; intent matching.
- 02Modular intents for rapid updates.
WHAT CHANGED
- Snappy prototype for FAQs & utilities.
ARCHITECTURE
The system follows a pipeline architecture from raw text input to predicted intent:
1. User Input → Preprocessing: Raw user text is normalized: lowercasing, tokenization (splitting into words), and stemming (reducing words to root forms with the Porter stemmer). This reduces vocabulary size and improves generalization.
2. Preprocessing → Feature Extraction: Processed tokens are converted into numerical representations using bag-of-words (BoW) encoding. Each word in the training vocabulary becomes a binary feature (1 if present, 0 if absent), creating a fixed-length vector that represents the input sentence.
3. Feature Vector → Neural Network: The bag-of-words vector feeds into a feedforward neural network trained to classify intents. The network learns to map word patterns to intent categories through supervised learning on the labeled training corpus.
4. Intent Prediction → Response Generation: The model outputs a probability distribution over all possible intents. The highest-probability intent is selected, and the system retrieves a corresponding response from the JSON corpus. Responses can be randomized from a pool to add variety.
JSON Corpus Structure: The training data is organized as intents, each containing example patterns (user inputs) and responses. So updates are quick: a new intent means extending the JSON file and retraining.
Data flow
click a stage
JSON file contains intent tags, example patterns, and response templates. Parser extracts training data and builds vocabulary.
COMPONENT
Training PipelineLoads JSON corpus, generates training data, performs data augmentation, and trains the neural network. Saves model weights and vocabulary for deployment.
Decisions, with the cost of each.
A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.
Classical NLP over Transformers
I used bag-of-words and a small neural network instead of BERT/GPT to keep compute low. For a lightweight deployment, a transformer would be overkill.
AT THE COST OF
Gives up semantic understanding of complex queries for speed and simplicity. Works well for domain-specific intents with clear patterns but struggles with highly ambiguous or context-dependent inputs.
Stemming vs. Lemmatization
I used the Porter stemmer, not lemmatization. Stemming is faster and needs no POS tagging, so the pipeline stays simpler and the vocabulary still shrinks.
AT THE COST OF
Stemming can be overly aggressive (e.g., 'university' → 'univers'), potentially merging semantically different words. Fine for intent classification, where exact word forms matter less than the pattern.
Modular JSON corpus architecture
Intents live as standalone JSON objects, not hardcoded logic, so someone non-technical can change the bot by editing JSON without touching code.
AT THE COST OF
Requires retraining when intents change, adding a deployment step.
Confidence threshold with fallback
A confidence score tells me when the model is unsure. Low-confidence predictions get a generic fallback instead of a possibly wrong answer.
AT THE COST OF
May miss some valid queries that fall outside training distribution. In anything customer-facing, admitting 'not sure' beats a wrong answer.
The part that mattered.
The code that did the work.
import nltk
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
def preprocess(sentence):
"""
Tokenize and stem input sentence.
Returns list of stemmed word tokens.
"""
# Tokenize into words
tokens = nltk.word_tokenize(sentence.lower())
# Remove punctuation and stem each word
stemmed_words = [
stemmer.stem(word)
for word in tokens
if word.isalnum()
]
return stemmed_words
# Example
# Input: "What's the weather forecast?"
# Output: ['what', 'the', 'weather', 'forecast']
def bag_of_words(tokenized_input, vocabulary):
"""
Convert tokenized input to binary bag-of-words vector.
vocabulary: list of all unique words from training data
"""
bag = [0] * len(vocabulary)
for token in tokenized_input:
if token in vocabulary:
idx = vocabulary.index(token)
bag[idx] = 1
return bag
# Example
# vocabulary = ['hello', 'weather', 'forecast', 'goodbye']
# Input tokens: ['weather', 'forecast']
# Output: [0, 1, 1, 0]
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Dropout
def build_model(input_size, num_intents):
"""
Build feedforward neural network for intent classification.
"""
model = Sequential([
Dense(128, input_shape=(input_size,), activation='relu'),
Dropout(0.5),
Dense(64, activation='relu'),
Dropout(0.5),
Dense(num_intents, activation='softmax')
])
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy']
)
return model
# Architecture:
# Input Layer: Bag-of-words vector (e.g., 200 features)
# Hidden Layer 1: 128 neurons, ReLU activation
# Dropout: 50% to prevent overfitting
# Hidden Layer 2: 64 neurons, ReLU activation
# Dropout: 50%
# Output Layer: Softmax over intent classes
import numpy as np
def predict_intent(user_input, model, vocabulary, intents, threshold=0.7):
"""
Predict intent from user input with confidence thresholding.
"""
# Preprocess input
tokens = preprocess(user_input)
bow = bag_of_words(tokens, vocabulary)
bow_array = np.array([bow])
# Get prediction probabilities
predictions = model.predict(bow_array)[0]
max_prob = np.max(predictions)
# Check confidence threshold
if max_prob < threshold:
return {
'intent': 'unknown',
'confidence': max_prob,
'response': "I'm not sure I understand. Could you rephrase?"
}
# Get highest probability intent
intent_idx = np.argmax(predictions)
predicted_intent = intents[intent_idx]
return {
'intent': predicted_intent['tag'],
'confidence': float(max_prob),
'response': np.random.choice(predicted_intent['responses'])
}
{
"intents": [
{
"tag": "greeting",
"patterns": [
"Hi",
"Hello",
"Hey there",
"Good morning",
"What's up"
],
"responses": [
"Hello! How can I help you today?",
"Hi there! What can I do for you?",
"Hey! What brings you here?"
]
},
{
"tag": "weather",
"patterns": [
"What's the weather like?",
"Tell me the forecast",
"Is it going to rain?",
"Weather forecast please"
],
"responses": [
"I can help with weather info! What city?",
"Sure, which location do you need weather for?"
]
},
{
"tag": "goodbye",
"patterns": [
"Bye",
"See you later",
"Goodbye",
"Thanks, that's all"
],
"responses": [
"Goodbye! Have a great day!",
"See you later!",
"Thanks for chatting!"
]
}
]
}
LEARNED
Intent classification quality depends heavily on training data diversity
The model performs well only when training patterns cover the variety of ways users express the same intent. Data augmentation through paraphrasing and synonym replacement helped expand coverage without manual labeling.
Preprocessing consistency is critical between training and inference
Early bug: training data was stemmed but inference input wasn't, causing poor predictions. I moved preprocessing into shared functions and checked that the exact same pipeline runs in training and deployment. Saved preprocessing configuration alongside model weights.
Bag-of-words loses word order information
BoW treats 'not good' and 'good' identically if both words appear, missing negation. For this project scope it was acceptable, but production systems need n-grams or sequential models (RNN/LSTM) to capture context. I found this one testing edge cases.
Confidence thresholding significantly improves user experience
Without confidence checks, the bot would confidently return wrong answers for out-of-domain queries. A 0.7 threshold with fallback responses cut down user frustration. Better to say 'I don't know' than mislead. I picked the threshold by testing values on the validation set.
Neural networks vs. classical ML for intent classification
Experimented with both neural networks and traditional classifiers (SVM, Naive Bayes). The neural network handled sparse BoW vectors better and generalized better to unseen patterns, though it's more complex. The non-linear activation functions helped learn subtle pattern combinations that linear models missed.
Keeping the pieces separate made iteration fast
Separating intent definitions (JSON), preprocessing logic (Python module), and model architecture (separate file) made experiments faster. I could swap models or add intents without refactoring, which helped a lot when trying different network architectures and preprocessing.