How To Use Stanza By Stanford NLP Group (With Python Code)
Stanza is a Python natural language analysis library created by the Stanford NLP group. It is a collection of NLP tools that can be used to create neural network pipelines for text analysis. It supports functionalities like tokenization, multi-word token expansion, lemmatization, part-of-speech (POS), morphological features tagging, dependency parsing, named entity recognition(NER), and sentiment analysis. It uses Universal Dependencies to provide consistent annotations of grammar in over 60 human languages. Additionally, it provides a Python interface to the CoreNLPJava package. This can be used to inherit additional functionalities like constituency parsing, coreference resolution, and linguistic pattern matching. Stanza’s neural network NLP pipeline Installation Pip: pip install stanza Conda: conda install -c stanfordnlp stanza From Source: git clone https://github.com/stanfordnlp/stanza.git cd stanza pip install -e . Creating a Pipeline Stanza provides a plethora of pre-trained NLP models for 66 human languages that we can make use of. Downloading a pre-trained model and creating a pipeline is as easy as: stanza.download('en') nlp = stanza.Pipeline('en') Specifying the Model Package and download directory By default, this downloads the default package and all processors for the language specified, English in our case, to the home directory. A language may have multiple packages trained on different datasets, for example, English has four available packages – ewt[defualt], gum, lines, and partut. A list of all available languages and corresponding packages can be found here. To explicitly choose the desired package we use the package argument. And to change the download location of the model we make use of the model_dir argument. stanza.download('en', model_dir = ‘/models/english/’, package = ‘partut’) nlp = stanza.Pipeline('en', package = 'partut') Specifying the Processors Depending on the use-case one might need to specify a set of processors and the package to fetch the different processors from. There are two ways for specifying the processors argument: using a string of comma-separated processorsusing a dictionary of processor-package pairs 1. Only downloading the required processors: To download only the required processors we can use a list of processors string like shown below stanza.download('hi', processors='tokenize,pos') nlp = stanza.Pipeline('hi', processors='tokenize,pos') Downloads and loads the default tokenize (TokenizeProcessor) and pos (POSProcessor) processors for Hindi. 2. Specifying package names for the processors: Choosing the package name for processors can be done using the package argument. stanza.download('it', processors='tokenize,mwt', package='twittiro') nlp = stanza.Pipeline('it', processors='tokenize,mwt', package='twittiro') Downloads and initializes the tokenize (TokenizeProcessor) and mwt (MWTProcessor) trained on the twittiro dataset for Italian. We may need to specify the package for one or a few processors and keep the default package for the rest, this can be achieved using the dictionary-based processor’s argument. This example shows how to download and load the NERProcessor trained on the Dutch CoNLL02 dataset, but use the default package for all other processors for Dutch. stanza.download('nl', processors={'ner': 'conll02'}) nlp = stanza.Pipeline('nl', processors={'ner': 'conll02'}) For a more granular control over the package names for the processors, we can set the package argument to None and use the dictionary-based processor’s argument to specify the package name for each process. The example shows how to use a GSD TokenizeProcessor, an HDT POSProcessor,  a CoNLL03 NERProcessor, and a default LemmaProcessor for German. processor_dict = { 'tokenize': 'gsd', 'pos': 'hdt', 'ner': 'conll03', 'lemma': 'default' } stanza.download('de', processors=processor_dict, package=None) nlp = stanza.Pipeline('de', processors=processor_dict, package=None) Creating & Overwriting Processors In version 1.1, Stanza added the ability to create new Processors and to overwrite existing ones using the decorator @register_processor_variant. from stanza.pipeline.processor import Processor, register_processor, register_processor_variant @register_processor("lowercase") class LowercaseProcessor(Processor): ''' Processor that lowercases all text ''' _requires = set(['tokenize']) _provides = set(['lowercase']) def __init__(self, config, pipeline, use_gpu): pass def _set_up_model(self, *args): pass def process(self, doc): doc.text = doc.text.lower() for sent in doc.sentences: for tok in sent.tokens: tok.text = tok.text.lower() for word in sent.words: word.text = word.text.lower() return doc nlp = stanza.Pipeline(lang='en', processors='tokenize,lowercase') doc = nlp("Question answering is a task where a sentence or sample of text is provided from which questions are asked and must be answered.") s =[] for sentence in doc.sentences: for word in sentence.words: s.append(word.text) print(" ".join(s)) Annotating a Document After a pipeline has been created, we can annotate a string/document by simply passing it to the Pipeline object. stanza.download('hi') hi_nlp = stanza.Pipeline('hi') hindi_doc = hi_nlp("प्रश्न का उत्तर देना एक ऐसा कार्य है जहाँ एक वाक्य या पाठ का नमूना प्रदान किया जाता है जहाँ से प्रश्न पूछे जाते हैं और उसका उत्तर दिया जाना चाहिए।") Printing each word with its lemma and POS tag: for sentence in hindi_doc.sentences: for word in sentence.words: print("{:12s}\t{:12s}\t{:6s}".format(word.text,word.lemma, word.pos)) Printing all the named entities with their dependencies: for sentence in hindi_doc.sentences: print(sentence.ents) print(sentence.dependencies) Dependencies visualized on the online demo available here: http://stanza.run/ Biomedical & Clinical Models Stanza also provides packages that support syntactic analysis and named entity recognition (NER) on both English biomedical literature and clinical note text. Offered packages include: 2 biomedical syntactic analysis pipelines, trained with human-annotated treebanks1 clinical syntactic analysis pipeline, trained with silver data8 biomedical NER models augmented with contextualized representations2 clinical NER models, including one specialized in radiology reports. A list of all available biomedical packages with their performance is available here The Stanza biomedical models can be used in the same way as the normal NLP models. The example below shows the code for downloading and initializing the i2b2 clinical NER model and annotating the various entities in a clinical note text. stanza.download('en', package='mimic', processors={'ner': 'i2b2'}) nlp = stanza.Pipeline('en', package='mimic', processors={'ner': 'i2b2'}) doc = nlp('The patient had a dry cough and fever, they were treated with Paracetamol.') # print out the entities for ent in doc.entities: print(f'{ent.text}\t{ent.type}') GitHubDocumentationCollab Notebook
Stanza is a Python natural language analysis library created by the Stanford NLP group. It is a collection of NLP tools that can be used to create neural network pipelines for text analysis. It supports functionalities like tokenization, multi-word token expansion, lemmatization, part-of-speech (POS), morphological features tagging, dependency parsing, named entity recognition(NER), and sentiment analysis. […]
Stanza is a collection of NLP tools that can be used to create neural pipelines for text analysis by the Stanford NLP group
["Natural Language Processing", "NLP", "NLP libraries", "python visualize neural network", "text analytics", "tokenization"]
["AI", "neural network", "Natural Language Processing", "sentiment analysis", "tokenization", "Git", "NLP", "NLP libraries", "Python", "python visualize neural network", "text analytics", "programming_languages:Python", "GitHub", "R", "Java"]
["AI", "neural network", "NLP", "sentiment analysis", "Python", "R", "Java", "Git", "GitHub", "programming_languages:Python"]
A machine learning enthusiast with a knack for finding patterns. In my free time, I like to delve into the world of non-fiction books and video essays.