What is a task in this context?

• Task is a data generating distribution. A distribution over inputs, a distribution over labels given inputs and a loss function.

  

• Goal: Given training data sample from those distribution and minimize the loss function in a way that generalizes to new data points sampled from those distributions

• Learning a task means minimizing the loss function and training over the data to learn the parameters of the network

How to train a model to solve multiple tasks?

• We can train a model to do multiple task. We use a task descriptor zi to condition in that.
  • How we condition zi affects how parameters are shared across tasks.
    • If you observe negative transfer meaning task performing poorly we should try to share less. Negative transfer mean independent network work better at times.
      • Why does this happen?
        • optimization challenges
          • caused by cross-task interference
          • tasks may learn at different rates
        • limited representational capacity
          • multi-task networks often need to be much larger than their single-task counterparts
    • If we observe overfitting we need to share between tasks

What is the objective of the multi-task learning?

• Objective of multi task learning is basically to minimize the loss function across all the tasks given the network parameter and the training datasets
  • This could be changes to a weighted version given the priority of the task
  • Objective also impacts the loss function weights. This depends on condition like task priority, data etc
• Task Descriptor
  • Example
    • one-hot encoding of the task index
    • personalization: user features/attributes
    • language description of the task
    • formal specifications of the task
• Conditioning
  • Vanilla is multiple training within a single network with no parameter shared but with different prediction heads and then using multiplicative gating across heads
  • Extreme Concatenate the task descriptor with input or with activation function
  • Split network parameter θ into θsh to θi, objective is to minimize loss function given the network parameters and the dataset
  • Choosing to condition task descriptor = Choosing how and where to share parameters
  • Concat, Additive, multiplicative, cross-stitch, attention etc..

Basic Algorithm

• Sample mini-batch of tasks
• Sample mini-batch datapoints for each task
• Compute loss on the mini-batch
• Back propagate loss to compute gradient
• Apply gradient with you a neural net optimizer like Adam or SGD etc
  • Note: This ensures tasks are sampled uniformly regardless of data quantities
  • For regression problems ensure task labels are at same scale

Key Design Decisions