自然语言处理NLP系列（一）——文本预处理

自然语言处理NLP系列（一）——文本预处理

1 英文文本预处理

1.1 个人认为英文文本预处理分两步就可

第一步，将文本中含有的常用标点符号用空格替换掉，这么做的原因是英文中标点和单词之间是没有空格，后面用空格作为分隔符分词的时候就容易造成标点和单词在一起造成单词就不是原来的词了；
第二步，形成停用词表，去除停用词，停用词主要包括语气词，句子连接词等对文本的语义理解作用不大的词汇。

1.2 代码实现

stopwordsFile = "./stopwords"
 # 构建停用词表
 if not os.path.isfile(stopwordsFile):  # 判断文件是否存在
     raise ValueError("Can't find a stopwords file at path '{}'.".format(stopwordsFile))
 stopwords = []
 with open(stopwordsFile, "r") as f:
     while True:
         line = f.readline()
         if not line:
             break
         self.stopwords.append(line.strip("\n").strip())
import re
text = ""
# 第一步，非英文字符用空格代替
text = re.sub(r"'s", " 's", text)
text = re.sub(r"'ve", " 've", text)
text = re.sub(r"n't", " n't", text)
text = re.sub(r"'re", " 're", text)
text = re.sub(r"'d", " 'd", text)
text = re.sub(r"'ll", " 'll", text)
text = re.sub(r"[^A-Za-z]", " ", text)
# 第二步，转小写
do_lower_case = True
if do_lower_case:
    text = text.strip().lower()
# 第三步，分词，去停用词
stopwords = []
tokens = text.split()
text_words = [token for token in tokens if token not in stopwords]
print(text_words)

附录——常见英文停用词

ZT
ZZ
a
able
about
above
abst
accordance
according
accordingly
across
act
actually
added
adj
adopted
affected
affecting
affects
after
afterwards
again
against
ah
ain
all
allow
allows
almost
alone
along
already
also
although
always
am
among
amongst
an
and
announce
another
any
anybody
anyhow
anymore
anyone
anything
anyway
anyways
anywhere
apart
apparently
appear
appreciate
appropriate
approximately
are
area
areas
aren
aren
arent
arise
around
as
aside
ask
asked
asking
asks
associated
at
auth
available
away
awfully
b
back
backed
backing
backs
be
became
because
become
becomes
becoming
been
before
beforehand
began
begin
beginning
beginnings
begins
behind
being
beings
believe
below
beside
besides
best
better
between
beyond
big
biol
both
brief
briefly
but
by
c
ca
came
can
cannot
cant
case
cases
cause
causes
certain
certainly
changes
clear
clearly
co
com
come
comes
concerning
consequently
consider
considering
contain
containing
contains
corresponding
could
couldn
couldnt
course
currently
d
date
definitely
describe
described
despite
did
didn
differ
different
differently
discuss
do
does
doesn
doing
don
done
down
downed
downing
downs
downwards
due
during
e
each
early
ed
edu
effect
eg
eight
eighty
either
else
elsewhere
end
ended
ending
ends
enough
entirely
especially
et
et-al
etc
even
evenly
ever
every
everybody
everyone
everything
everywhere
ex
exactly
example
except
f
face
faces
fact
facts
far
felt
few
ff
fifth
find
finds
first
five
fix
followed
following
follows
for
former
formerly
forth
found
four
from
full
fully
further
furthered
furthering
furthermore
furthers
g
gave
general
generally
get
gets
getting
give
given
gives
giving
go
goes
going
gone
good
goods
got
gotten
great
greater
greatest
greetings
group
grouped
grouping
groups
h
had
hadn
happens
hardly
has
hasn
have
haven
having
he
hed
hello
help
hence
her
here
hereafter
hereby
herein
heres
hereupon
hers
herself
hes
hi
hid
high
higher
highest
him
himself
his
hither
home
hopefully
how
howbeit
however
hundred
i
id
ie
if
ignored
im
immediate
immediately
importance
important
in
inasmuch
inc
include
indeed
index
indicate
indicated
indicates
information
inner
insofar
instead
interest
interested
interesting
interests
into
invention
inward
is
isn
it
itd
its
itself
j
just
k
keep
keeps
kept
keys
kg
kind
km
knew
know
known
knows
l
large
largely
last
lately
later
latest
latter
latterly
least
less
lest
let
lets
like
liked
likely
line
little
ll
long
longer
longest
look
looking
looks
ltd
m
made
mainly
make
makes
making
man
many
may
maybe
me
mean
means
meantime
meanwhile
member
members
men
merely
mg
might
million
miss
ml
mon
more
moreover
most
mostly
mr
mrs
much
mug
must
my
myself
n
na
name
namely
nay
nd
near
nearly
necessarily
necessary
need
needed
needing
needs
neither
never
nevertheless
new
newer
newest
next
nine
ninety
no
nobody
non
none
nonetheless
noone
nor
normally
nos
not
noted
nothing
novel
now
nowhere
number
numbers
o
obtain
obtained
obviously
of
off
often
oh
ok
okay
old
older
oldest
omitted
on
once
one
ones
only
onto
open
opened
opening
opens
or
ord
order
ordered
ordering
orders
other
others
otherwise
ought
our
ours
ourselves
out
outside
over
overall
owing
own
p
page
pages
part
parted
particular
particularly
parting
parts
past
per
perhaps
place
placed
places
please
plus
point
pointed
pointing
points
poorly
possible
possibly
potentially
pp
predominantly
present
presented
presenting
presents
presumably
previously
primarily
probably
problem
problems
promptly
proud
provides
put
puts
q
que
quickly
quite
qv
r
ran
rather
rd
re
readily
really
reasonably
recent
recently
ref
refs
regarding
regardless
regards
related
relatively
research
respectively
resulted
resulting
results
right
room
rooms
run
s
said
same
saw
say
saying
says
sec
second
secondly
seconds
section
see
seeing
seem
seemed
seeming
seems
seen
sees
self
selves
sensible
sent
serious
seriously
seven
several
shall
she
shed
shes
should
shouldn
show
showed
showing
shown
showns
shows
side
sides
significant
significantly
similar
similarly
since
six
slightly
small
smaller
smallest
so
some
somebody
somehow
someone
somethan
something
sometime
sometimes
somewhat
somewhere
soon
sorry
specifically
specified
specify
specifying
state
states
still
stop
strongly
sub
substantially
successfully
such
sufficiently
suggest
sup
sure
t
take
taken
taking
tell
tends
th
than
thank
thanks
thanx
that
thats
the
their
theirs
them
themselves
then
thence
there
thereafter
thereby
thered
therefore
therein
thereof
therere
theres
thereto
thereupon
these
they
theyd
theyre
thing
things
think
thinks
third
this
thorough
thoroughly
those
thou
though
thoughh
thought
thoughts
thousand
three
throug
through
throughout
thru
thus
til
tip
to
today
together
too
took
toward
towards
tried
tries
truly
try
trying
ts
turn
turned
turning
turns
twice
two
u
un
under
unfortunately
unless
unlike
unlikely
until
unto
up
upon
ups
us
use
used
useful
usefully
usefulness
uses
using
usually
uucp
v
value
various
ve
very
via
viz
vol
vols
vs
w
want
wanted
wanting
wants
was
wasn
way
ways
we
wed
welcome
well
wells
went
were
weren
what
whatever
whats
when
whence
whenever
where
whereafter
whereas
whereby
wherein
wheres
whereupon
wherever
whether
which
while
whim
whither
who
whod
whoever
whole
whom
whomever
whos
whose
why
widely
will
willing
wish
with
within
without
won
wonder
words
work
worked
working
works
world
would
wouldn
www
x
y
year
years
yes
yet
you
youd
young
younger
youngest
your
youre
yours
yourself
yourselves
z
zero