# BART pre-training?

**URL:** <https://discuss.huggingface.co/t/bart-pre-training/549>\
**Category:** Beginners\
**Created:** [July 31, 2020, 8:59pm UTC](https://discuss.huggingface.co/t/bart-pre-training/549 "2020-07-31T20:59:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\
**Post date:** [July 31, 2020, 8:59pm UTC](https://discuss.huggingface.co/t/bart-pre-training/549/1 "2020-07-31T20:59:56Z")

</div>

Hi,  
How can I pre-train Bart with our own dataset? It seems that the script examples/language-modeling/run\_language\_modeling.py doesn’t support it yet. Thanks.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 1, 2020, 4:38am UTC](https://discuss.huggingface.co/t/bart-pre-training/549/2 "2020-08-01T04:38:49Z")

</div>

Hi @cahya, BART pre-training is not yet available in transformers. You can find the denoising dataset [here](https://github.com/pytorch/fairseq/blob/master/fairseq/data/denoising_dataset.py#L78) in the fairseq repo, and try to use it;

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\
**Post date:** [August 1, 2020, 12:23pm UTC](https://discuss.huggingface.co/t/bart-pre-training/549/3 "2020-08-01T12:23:39Z")

</div>

Thanks for the info and the link to the denoising dataset. Maybe @sshleifer can tell us also his experience with BART and transformers?  
And how about T5? I see that you have already several T5 models, can we pre-train T5 with our own dataset using transformers?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 1, 2020, 2:06pm UTC](https://discuss.huggingface.co/t/bart-pre-training/549/4 "2020-08-01T14:06:09Z")

</div>

Adding both of these tasks (T5 and BART pre-training) is in my todo list. Might take some time though.  
If you are able to create the span masking code for T5 then you can easily pre-train T5 with Transformers.

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\
**Post date:** [August 1, 2020, 2:27pm UTC](https://discuss.huggingface.co/t/bart-pre-training/549/5 "2020-08-01T14:27:39Z")

</div>

Great, I’ll try it

---

<div class="post-metadata">

**Author:** ![Arielkanevsky](https://avatars.discourse-cdn.com/v4/letter/a/fbc32d/32.png) [@Arielkanevsky](https://discuss.huggingface.co/u/Arielkanevsky)\
**Post date:** [August 5, 2023, 4:59am UTC](https://discuss.huggingface.co/t/bart-pre-training/549/6 "2023-08-05T04:59:33Z")

</div>

Any updated on this?