【Hugging Face】Ep.3 前往Datasets掏金趣

鎖定目標，縮小範圍

玩轉資料集

!pip install datasets

from datasets import load_dataset_builder
ds_builder = load_dataset_builder("imdb")

檢查資料集資訊

from datasets import load_dataset_builder
ds_builder = load_dataset_builder("imdb")

print(ds_builder.info.description)
# Large Movie Review Dataset.

print(ds_builder.info.features)
# This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well.
{'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}

索引值操作

from datasets import load_dataset

# 載入訓練的資料集
ds = load_dataset("imdb", split='train')
# 第一列
ds[0]
# 最後一列
ds[-1]

過濾

ds1 = ds.filter(lambda x: 'U.S' in x['text'] and len(x['text']) < 500)

ds1[:3]
{
'text': [
  'It is not un-common to see U.S. re-makes of foreign movies that fall flat on their face, but here is the flip side!!! This is an awful re-make of the U.S. movie "Wide Awake" by the British!<br /><br />"Wide Awake" is strange but entertaining and funny! "Liam" on the other hand is just strange. I must give credit to "Liam" for one thing, and that is making it clear that I made the right choice in changing my religion!',
  'I saw this movie on Comedy Central a few times. This movie was pretty good. It\\'s an interesting adventure with the life of Sunny Davis, who is arranged to marry the king of Ohtar, so that the U.S. can get an army base there to balance power in the Middle East. Some good jokes, including "Sunnygate." I also just loved the ending theme. It gave me great political spirit. Ten out of ten was my rating for this movie.',
  '"Antwone Fisher" tells of a young black U.S. Navy enlisted man and product of childhood abuse and neglect (Luke) whose hostility toward others gets him a stint with the base shrink (Washington) leading to introspection, self appraisal, and a return to his roots. Pat, sanitized, and sentimental, "Antwone Fisher" is a solid feel-good flick about the reconciliation of past regrets and closure. Good old Hollywood style entertainment family values entertainment with just a hint of corn. (B)'],
'label': [0, 1, 1]
}

更多的操作方式

結語

更多關於【Hugging Face 系列】…

• 【Hugging Face】Ep.1 平凡人也能玩的起的AI平台
• 【Hugging Face】Ep.2 初探新手村, 打造專屬的翻譯官
• 【Hugging Face】Ep.3 前往Datasets掏金趣
• 【Hugging Face】Ep.4 文字與模型的轉換器Tokenizer
• 【Hugging Face】Ep.5 文字世界中的超能力語言英雄(Named Entity Recognition)
• 【Hugging Face】Ep.6 解決問題的專業級破關知識家(Question Answering)
• 【Hugging Face】Ep.7 以文生圖的魔幻製圖師(Text To Image)
• 【Hugging Face】Ep.8 專業的標準化推理家(Inference API)