Neural Architecture Search and AutoML

The Problem AutoML Solves

Designing neural architectures and tuning hyperparameters is time-consuming and requires deep expertise. AutoML automates these processes.

Hyperparameter Optimization with Optuna

python
import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
def objective(trial):
    # Define hyperparameter search space
    n_estimators = trial.suggest_int('n_estimators', 10, 300)
    max_depth = trial.suggest_int('max_depth', 2, 32, log=True)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 10)
    
    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split
    )
    
    score = cross_val_score(model, X_train, y_train, cv=3, scoring='accuracy')
    return score.mean()
Run optimization
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"Best params: {study.best_params}")

Distributed Tuning with Ray Tune

python
from ray import tune
from ray.tune.schedulers import ASHAScheduler
import torch.nn as nn
def train_model(config):
    model = nn.Sequential(
        nn.Linear(784, config["hidden_size"]),
        nn.ReLU(),
        nn.Dropout(config["dropout"]),
        nn.Linear(config["hidden_size"], 10)
    )
    # Training loop...
    tune.report(accuracy=test_accuracy)
scheduler = ASHAScheduler(max_t=100, grace_period=10)
results = tune.run(
    train_model,
    config={
        "hidden_size": tune.choice([64, 128, 256, 512]),
        "dropout": tune.uniform(0.1, 0.5),
        "lr": tune.loguniform(1e-4, 1e-1)
    },
    scheduler=scheduler,
    num_samples=50
)

AutoGluon for Tabular Data

python
from autogluon.tabular import TabularPredictor
predictor = TabularPredictor(label='target').fit(
    train_data,
    time_limit=3600,  # 1 hour
    presets='best_quality'
)
predictions = predictor.predict(test_data)
leaderboard = predictor.leaderboard()

Neural Architecture Search (NAS)

python
Using NAS with a search space definition
search_space = {
    "num_layers": [2, 3, 4, 5],
    "hidden_dim": [64, 128, 256],
    "activation": ["relu", "gelu", "silu"],
    "use_attention": [True, False]
}

When to Use AutoML

Baseline comparison for new datasets

When ML expertise is limited

Time-constrained projects

Tabular data tasks