@Title: Challenges for Information Extraction
@File: mavir01
@Participants: GRI, Ralph, (man, C, 3, professor, lecturer, New York)
@Date: 16/11/2006
@Place: Madrid
@Situation: conference (I Jornadas MAVIR), conference room at university, not hidden, observing researcher
@Topic: Current situation of information extraction research
@Source: MAVIR
@Class: formal in natural context, conference, monologue
@Length: 1 h 07' 39"
@Words: 9113
@Acoustic_quality: A
@Transcriber: M. Garrote
@Revisor: L. Campillos, M. Garrote
@Comments:
GRI thank you Antonio / and thank you to the / entire MAVIR Network / for the invitation 00:06
GRI &ah / can everybody hear me in the back ? 00:09
GRI it's ok ? 00:10
GRI &ah / one of the dangers of having an introduction like this / and being told / how many years you've worked already in computation linguistics // is to be asked of it and / why didn't you get further by now ? 00:22
GRI after forty years we've learned / things go rather slowly / that we make progress / but progress is never as fast as / we predicted that it will be 00:33
GRI looking ahead / we think in five years / everything will be wonderful 00:37
GRI and now I say / well / maybe my children / or my grandchildren / will solve all these problems 00:43
GRI one of the nice things about / natural language processing is a problem // is that is so rich / so deep // that / you can / peel off one layer of problems // get some level of solutions // but there will still be many layers / left for the next rank of students to solve 01:02
GRI ok // so let me talk over about / the challenges for information extraction 01:09
GRI &ah / I present both / some very / general overview / and then some technical details about various projects we've worked on 01:20
GRI so let me begin by just explaining what I mean by information extraction 01:25
GRI so identifying instances / of important entities / relations and events / from unstructured texts 01:33
GRI ok // so for example / &ah / this is one / we've worked on enough for a long time // identifying people who are hired and fired / by various companies 01:48
GRI so we have the sentence // George Garret / forty years old / president of London-based European Information Services / was appointed chief executive officer of Nielsen Marketing 01:59
GRI and somebody comes and says // ok // we want to read all the newspapers // and keep track of who's been hired or fired 02:06
GRI so we'd like to produce = 02:08
GRI see if my mouse is awake 02:10
GRI no 02:10
GRI not awake up this morning 02:12
GRI &ah / we'd like to produce a database / which lists position // and the company / and the location / and the person / and whether they got / left the job / or they got into the job 02:25
GRI cause that's a typical example of what we / mean by doing information extraction 02:30
GRI so to make a contrast / xxx / also hear about information retrieval // from the web / contrast between information extraction / and / information retrieval 02:42
GRI so / in information extraction / the operations are based on and then return normalized values like dates // or entities 02:52
GRI so by an entity I mean not just the reference / but / if we have several references to the same person / to recognize that / as / the single particular person // and most important relations between entities / or between a company / and a person // between a person and a location 03:10
GRI so information extraction has to be adapted to specific tasks 03:15
GRI somebody will come and say / we're interested in hirings and firings 03:18
GRI or we are interested in / talks of universities 03:22
GRI or we are interested in / a task on cities or something 03:26
GRI and the system has to be customized for that particular task 03:30
GRI in contrast / information retrieval / basically based on terms // just on [/] on differentiated tokens 03:40
GRI and we would turn documents or passages right than specific facts 03:44
GRI and the benefit is that it's a general technology // something that doesn't have to be customized for a particular task 03:51
GRI but the power I'd like to suggest of information extraction is that there's some questions / which are really hard to answer using just / Google 04:01
GRI so if you wanted to get an answer to where has Condoleezza Rice been in the last month // you might have to Google for her name // see which articles refer to her / locations // and then slowly xxx the database might take you ten or twenty minutes / to answer this question 04:20
GRI whereas with information extraction if we're interested in tracking people and their locations // this is a plausible question we might pose directly 04:28
GRI hhh {%act: cough} or what terrorist attacks occurred in Europe in 2004 04:34
GRI if you xxx in terrorist attacks in Europe in 2004 // you might not get very much 04:39
GRI 'cause articles might talk about England / and Spain / and various countries 04:44
GRI if you don't have relationships capture this part of retrieval // you're not going to do very well from term retrieval alone 04:51
GRI one way in which / information extraction might be used / this is a search tool / &ah as a complement for / sort of standard Google-based tools for doing web search 05:07
GRI the idea here is that people would be interested in particular domain and they'd come and say / we're interested in / &ah searching articles 05:14
GRI in this example I'll show in a moment about / disease outbreaks 05:19
GRI so we'll build the system for extracting relationships about disease outbreaks 05:25
GRI what disease / occurred when / and where / and so forth 05:28
GRI and then we pick the system // and we run it each day against the days news / using a web crawler up and down all the new sites 05:38
GRI retrieve the latest news // build the database // and then we provide access to the article through this database 05:47
GRI so / basic flow is the following // the web crawler // some filter to just get articles which are relevant to the task 05:56
GRI then build an extraction engine which builds a database 05:59
GRI on the other side somebody xxx with the browser // and would look something like this 06:05
GRI so what we have over here on the top // is / a database meant to look very much like Excell // has the same sort of capabilities as an Excell database 06:17
GRI which is the document date / the disease name / the time / the location / the country of the location / how many cases were reported / whether / they were sick or dead // and specific task description 06:32
GRI so then you can search this just like you would in a Excel database / to restrictions // and when you are interested in a particular article / you click on a row / and up comes / at the bottom of the screen / the corresponding passage 06:46
GRI so in this case it will say somebody click on dengue fever // and it says down below that the article it came from / why / state officials reported one additional recent case of dengue fever / and six cases that occur hhh {%act: onomatopoeia} 07:01
GRI ok / the extraction technology's not completely reliable 07:06
GRI so we get maybe / two thirds of correct information / three quarters of correct information / in the database 07:12
GRI but then / the database gets / immediately link to articles // so / even though you can't rely on the database per se it's been accurate / certain information // if you view it as a search tool is really / can be much more powerful search tool for particular topics // and a key word scheme / like / Google or Yahoo 07:35
GRI 'cause you wouldn't expect if you typed in / the search terms to something like Google that / two thirds or three quarters of the articles were irrelevant 07:43
GRI so / this sort of tools / for doing directed search have been applied in a number of areas // been applied in genomics 07:53
GRI that's one area where there's certainly a lot of money / in the United States / and perhaps here also 07:58
GRI and / &ah / researchers go with the money is xxx 08:04
GRI so there's been quite a bit of work / on extracting information in particular about gene protein interactions 08:10
GRI there's an enormous literature xxx by genomics researchers 08:14
GRI you simply / can't keep up with all the staff that's been published / to get to search for something // it's a major effort 08:20
GRI so being &ab [/] being able to pull out this specific relationships // we get a much more powerful search vehicle 08:28
GRI in / medical applications / there's a lot of demand for generating summaries for medical reports 08:35
GRI what fraction of / patients have this outcome / who xxx and so forth 08:40
GRI and some of this is now being done by information extraction 08:44
GRI also / in the idea of going with the money is / &ah / financial / information extraction is also being a major area 08:53
GRI so / in keeping with this idea that / I've been slaving at this for a long time // the other people have been slaving at this area for even longer 09:04
GRI so / although the idea of doing search from information extraction / seems like a / sort of timely and novel one // it's one which has been around now for // let's see // fifty eight so it's / almost half a century 09:19
GRI so / back in nineteen fifty eight / there was a presentation / by Zellig Harris in Washington / where they had a conference 09:28
GRI how should we be doing information retrieval / in a time when / there was very little online and / being online mean / taking an article / and typing it all into punch cards / and reading those punch cards in a reader 09:40
GRI and having a couple of articles // punch cards // people who remember // &ah -> / having a small collection of articles online / maybe a few hundred 09:52
GRI but at that time Zellig Harris succeeded in thinking / how could we automate this process / and have more powerful search vehicles // than just having keywords search 10:02
GRI so he talked / in this fifty eight paper / about the idea of taking a set of articles / discovering the main relationships which appeared in the articles 10:12
GRI he was interested in scientific / literature 10:15
GRI automatically indexing the articles / xxx work was then probably UNIVAC I or / old 7090 mainframe // indexing the articles // and then doing retrieval based upon relationships 10:29
GRI so it's taking us maybe half a century for the technology to catch up to this ideas // to have / the corpus [/] corpus frame methods / which can now / analyze large portions of texts reliably 10:45
GRI and / basically only over the last decade / have methods for discovering relationships from texts 10:53
GRI ok 10:56
GRI so -> / so much for history 10:58
GRI now / one of the challenges of information extraction at the moment / to understand there isn't progress / we need to appreciate very briefly / what the basic approach is / to information extraction 11:10
GRI and then the problems which arise / because of the complexities of language 11:14
GRI so expend a couple of minutes talking about / these problems // and then go through a number of &ah / areas of current research / and how they're trying to address these problems 11:25
GRI in a particular xxx / hhh {%act: cough} / survey or advertisement for what's going on at NYU // and so we'll have a &l [///] NYU symbol / xxx all torches will appear here and there // to show what we've been doing 11:37
GRI ok 11:40
GRI so the basic approach doing information extraction is very simple 11:46
GRI suppose to come back to the early application we were interested in figuring out // people who are hired or fired by companies 11:54
GRI ok? 11:57
GRI so / I give you that challenge 12:01
GRI maybe you / take this challenge up in Spanish 12:04
GRI think about five ways of saying somebody was hired or fired from a job 12:09
GRI ok? 12:11
GRI you can probably think of a couple of ways xxx here 12:14
GRI ok? 12:14
GRI this was a classroom where nobody's allowed to / give up assignments 12:18
GRI I have everybody sit down and write down / three or four patterns they could remember 12:22
GRI ok? 12:25
GRI and then we have some programmers here // we tell them to write some Perl program / or Python program // something which this nice pattern matching // and just run against / some newspapers // and see how all the pattern matches 12:39
GRI and / if some of the pattern match / we say / ok / we take the person / we put in this column // and we take the job / we put in this column // and we take the company / we put in this column // and we're all done 12:53
GRI and we would go have coffee / and you don't have to hear about all my problems 12:56
GRI well / this works // but it doesn't work very well 13:02
GRI people would discover after / hhh {%act: cough} / would discover after they try it // that they make it / five percent / ten percent recall they're really lucky 13:15
GRI so / why the simple pattern matching like this not work so well ? 13:19
GRI well / as I think we all know language is really very complicated // and all of the different problems of language / come forth in trying to do information extraction 13:29
GRI so / for example / the lots of different words / people who write for the Wall Street Journal / have to write articles like this everyday 13:42
GRI somebody was named to the job // somebody was appointed to the job // somebody was selected for the job 13:47
GRI so they are good at / finding new ways of saying the same thing 13:51
GRI they're paid to find new ways of saying the same thing 13:55
GRI so that makes it nice for readers / but it makes it more work for us as computational linguists 14:01
GRI then there're different constructs for providing the same information 14:08
GRI IBM named Fred as president // IBM announced the appointment of Fred as president // Fred / who was named president by IBM 14:17
GRI and so on and so forth 14:19
GRI ok then / people could be referred to in different ways 14:24
GRI so we can have George H. W. Bush / former president Bush / is something scored forty one / because he was a forty first president // so you / can differentiate / forty one from forty three 14:37
GRI ok? 14:38
GRI who's / in the office now ? 14:40
GRI ok so / all of these problems have to be addressed / beyond pattern matching in order to get reasonable information extraction 14:47
GRI then there are some ambiguities // so I present some in English / but there are probably comparable situations in Spanish 14:56
GRI so Fred's appointment as professor 14:59
GRI versus Fred's three o'clock appointment with the dean 15:04
GRI it's just a meeting // not [/] no job get started 15:08
GRI ok? 15:09
GRI so you can just look forward appointment / with somebody's name / and find xxx and say / &ah / somebody got a job 15:15
GRI a problem we had when we started doing the disease outbreak / extraction system // we had in mind that there would be outbreaks of typhoid / outbreaks of dengue // and then we run it against the day's newspaper // and the most common pattern we got was outbreaks of violence 15:34
GRI so we got all the tasks instead of all the diseases 15:37
GRI so lexical ambiguity becomes a problem 15:42
GRI then the structures aren't simple 15:45
GRI you'd like to have / person was appointed to job 15:49
GRI ok? 15:50
GRI so / we search through a few articles // and we found the following 15:54
GRI I don't know / if people can even figure out what the subject and object of the sentence are in this 16:00
GRI for the Federal Election Commission / Bush picked Justice Department employee / and former Fulton County Georgia Republican chairman / Hans von Spakovsky / for one of the three openings 16:15
GRI ok? 16:17
GRI so the / problem for the / person and the system / is picked / what ? 16:23
GRI or picked whom ? 16:24
GRI and you have to get through just this department employee hhh {%act: onomatopoeia} to get / von Spakovsky // as the object to picked 16:33
GRI ok? 16:34
GRI so simple pattern matching is not going to work 16:37
GRI we wouldn't need to do structure analysis in order to figure out what's going on 16:41
GRI even if we get through all of this // we have problems where we may have to go across sentences 16:50
GRI George Garrick has served as president of Sony for thirteen years 16:54
GRI the company announced his retirement effective next may 16:57
GRI ok? 16:58
GRI so you have to figure out what's his company / we can fill the database with / the company and him 17:04
GRI hhh {%act: cough} ok? 17:08
GRI hhh {%act: cough} excuse me one second 17:10
GRI {%com: drinks} so all of this means that we have a lot of work to do // in analyzing language in order to / be able to do / effective information extraction 17:29
GRI hhh {%act: cough} below I'll discussed a lot problems / we can group them roughly into two basic / types of problems 17:37
GRI collecting the patterns for a given relationship // and identifying the instances of these patterns in the text 17:45
GRI and / I'll begin by looking at the second problem // identifying the instances 17:51
GRI hhh {%act: cough} so / as I try to explain / with this example of von Spakovsky // it really doesn't work xxx do / extraction by just writing patterns which look for / sequences of tokens 18:11
GRI person pick name / is not going to work with // person pick just hhh {%act: onomatopoeia} / Hans von Spakovsky 18:23
GRI like you have just some way of figuring out / that / von Spakovsky is the object of picked 18:29
GRI so the patterns have to be stated at the structural level 18:34
GRI which means / as we understand that / before you can really do information extraction / you have to do a lot of linguistic analysis 18:41
GRI you have to identify names // and classify the names as people // and organizations and locations 18:48
GRI you have to figure out the syntactic structure // so we know what's the subject and what's the object hhh {%act: onomatopoeia} picked 18:56
GRI and we have to figure out coreference / so it is / the company and him we know what has been referred to 19:03
GRI and if the analysis is wrong at anyone of these stages / the pattern's not going to match 19:08
GRI so / what have people done over the last / decade or two decades in trying to address this problem of structure analysis ? 19:17
GRI well / people have broken it down into / different kinds of subtasks 19:21
GRI so named entities finding names // finding syntactic structure // finding coreference 19:27
GRI and people specialize each one of these problems 19:32
GRI building separate typically now / corpus frame models for doing each one of these tasks 19:37
GRI hhh {%act: cough} so people have / built large corpora annotated with names // large treebanks annotated with / syntactic relations // and even coreference / corpora 19:51
GRI and / after they've done this / started machine learning for different methods // they come and they give papers and say / look / we can get this wonderful / level of performance 20:03
GRI we can get ninety percent performance / for recognizing names 20:07
GRI and ninety percent accuracy for doing parsing 20:10
GRI and well / we can do coreference so well but / that's something for a children to work on 20:15
GRI so / we'd look 20:17
GRI ninety percent accuracy 20:18
GRI let's go out and let's sell our product 20:20
GRI well / you look back at the problem 20:23
GRI you see actually we've just / decomposed the problem into / name analysis // reference resolution relation tagging 20:31
GRI each one of these is ninety percent accurate 20:34
GRI let's say 20:35
GRI ok? 20:35
GRI maybe we are not xxx of everything 20:37
GRI but let's say everything is ninety percent accurate 20:39
GRI well // the end result is gonna be ninety percent / times ninety percent / times ninety percent // depending on how many of these components you / put together 20:52
GRI and so / with three components maybe we have seventy percent // maybe we can't sell our system anymore 20:59
GRI so / what do we do ? 21:03
GRI well / we can just / go home crying 21:06
GRI or we can look at the problem and say / well / we decomposed it // and in decomposing the problem we've looked at / trying to optimize each problem separately / each task separately 21:20
GRI so if looked separately / &ah finding the best names // finding the best relations // finding the best events and so forth 21:28
GRI and what we should do now / that we've decomposed these problems is take advantage of the interactions between the stages 21:35
GRI so / instead of making that xxx // try to / pick advantage / of all of these stages 21:42
GRI so what does that mean ? 21:45
GRI it means for example / preferring names / which allow for more coreference 21:49
GRI so / if you couldn't tell / if it was / the name was ABC // or ABCD 21:55
GRI but you find / some other articles xxx ABC // then most likely is gonna be the name mentioned more than once 22:03
GRI so / we did statistics like this // in which we looked at names which were only mentioned once // and names which were mentioned more than once 22:14
GRI and this graph shows &ah / if the name only appears once in an article 22:21
GRI and the chance that we got the name right is pretty low 22:24
GRI it's between forty five and sixty five percent 22:27
GRI but if the name is the same / as another mentioned which appeared somewhere else in the article / or in a different article // then we're much more confident that we got the name correct in this instance 22:38
GRI so if we can't tell whether it's von Spakovsky / or the name was just Spakovsky / and so forth 22:43
GRI if you see / somewhere else von Spakovsky // then you have much more confidence that you got the name correct 22:51
GRI hhh {%act: cough} &eh similar observation can be made about the connection between names and relations 22:58
GRI so / &ah the main fact to be seen from this graph / is the difference between / for each pair / of bars between the light purple and the dark purple 23:11
GRI so the light purple indicates / a name which appears in a relationship // like von Spakovsky / Fred's father / or something 23:20
GRI and the / purple / a name which shouldn't appear any relationships 23:26
GRI and basically what this graph tells us / let's see / we just look at the last bar // that / if it didn't participate in a relationship we only have a fifty percent chance / that we've got the name correct 23:39
GRI but if it participate in a relationship / we've got a ninety percent chance 23:43
GRI so / it's basically saying / if we've got the name right // then there's a much better chance / that we'll be able to identify relationship 23:51
GRI and so xxx backwards and say // if we've got a relationship / involving this name / then we probably identify this correctly as a person 24:00
GRI if we have some text where we have hhh {%act: onomatopoeia} / Fred's father // and we couldn't figure out whether hhh {%act: onomatopoeia} was a person or an organization or a location // you can probably tell from the fact that was Fred's father // and things on to screen here / that this was a person rather an organization 24:18
GRI so we try to use this sort of / relationship / between the different stages // and use in particular the constraints / that semantic relations / impose on the arguments 24:31
GRI so if we have hhh {%act: onomatopoeia} somebody's father // that means it has to be a person 24:35
GRI and we use these relationships to pick / preferred analysis 24:41
GRI so / how do we put that together ? 24:43
GRI or the basic idea is instead of with the name analyzer / analyze just / one possibility 24:49
GRI if the / analyzer is not sure what / the correct analysis is 24:55
GRI xxx generate multiple possibilities 24:58
GRI so-called / N-best choices 25:00
GRI so we can't tell whether the name is von Spakovsky or Spakovsky / we'll take both and will say // I'm not sure / what the analysis is 25:09
GRI let me pass two hypothesis on to the next stage 25:12
GRI then the next stage we'll re-rank this / based upon / the overall sentence structure / upon the relationships / upon coreference 25:20
GRI and we found / in this analysis by one of the students working in Chinese // that we could get a substantial reduction in the error rate 25:27
GRI just between / name analysis / coreference / and relations 25:34
GRI so the overall picture now look something like this // which start with the raw document // we've got a bunch of hypothesis for names / a bunch for coreference / a bunch for relations 25:48
GRI and once we've / put them all together // we go to this re-ranking model / which looks for global optimum 25:56
GRI and / by finding a global optimum across names and relations / and coreference // we're not able to get ninety percent / but maybe we can get eighty five percent accuracy instead of just seventy percent accuracy 26:10
GRI ok? 26:13
GRI so there's been several researches of work in this / &ah xxx view / &ah / professor Roth at / &ah Illinois also been working on this / in terms of a more standard optimization strategy 26:26
GRI so / I won't go into the / detail here // but I think this is a trend we're gonna see much more / instead of / optimizing the components separately // to have a situation now where we / to much credit extent use the components together // in order to break through this performance variants 26:45
GRI ok 26:48
GRI so / see that it's the main hope / in terms of improving overall performance // of course people will continue working on / coreference and names / and relations separately 26:58
GRI but I think it's gonna be the synthesis / of analysis which is going to / push us to our higher performance 27:05
GRI ok / let's take all one now and look at the other type of problem // the problem of collecting the patterns for a given relationship 27:12
GRI so / as I was saying a few minutes ago // there're lots of ways of expressing an event 27:21
GRI so this is / xxx / assassination of president Lincoln back in eighteen sixty five 27:28
GRI Booth assassinated Lincoln ? 27:31
GRI Lincoln was assassinated by Booth 27:33
GRI the assassination of Lincoln by Booth 27:36
GRI Booth went through with the assassination of Lincoln 27:39
GRI Booth murdered Lincoln 27:40
GRI Booth fatally shot Lincoln 27:42
GRI we can probably go through ten or twenty / or thirty or fifty / or maybe a hundred different ways / in which somebody can be / killed 27:50
GRI and again the situation is / if I / ask you to name / five ways of doing this // and say you can't go to coffee until you've found five ways // probably everybody will come up for five ways 28:06
GRI maybe / we have / almost hundred people and may have xxx data collection here // collect all the ways in which can we represent it 28:15
GRI but I think there still be a lot of ways missing 28:18
GRI and if I told people // you have to find a hundred ways of saying somebody was killed / or we can't get coffee // I think we'll have some very angry people in the audience 28:28
GRI so / what do we do about this ? 28:32
GRI this tail / the standard tail of the distribution which we see in some many linguistics' problems 28:39
GRI always comes to xxx / or we try to get good coverage 28:42
GRI so one thing we could do is / spend the rest of the afternoon reading newspapers 28:51
GRI ok? 28:53
GRI so / &ah we go and / collect El País // so whatever everybody reads here 29:00
GRI and hand out a copy 29:02
GRI everybody gets one day's copy of the newspaper 29:05
GRI we have some people here // &ah / Antonio // people who are expert in doing corpus annotation // so they will organize all the corpus annotated xxx / everybody will get a / marker // and have to mark / all the sentences saying somebody got hired or fired 29:21
GRI ok? 29:24
GRI and / we can / this maybe if someone paying for xxx but / you can everybody do this for a few days // you have to stay here for the rest of the week // and just mark / instances 29:35
GRI ok? 29:36
GRI well / again / some people might not be very happy with / doing this // 'cause I don't know if some people might prefer reading the newspaper 29:42
GRI so I'm not sure 29:43
GRI can we somehow / automate this process / in so get rid of all of this manual annotation ? 29:49
GRI and that's really what we're looking at / most of the rest of this talk 29:54
GRI how do we collect patterns more automatically ? 29:57
GRI in order to address this problem / I've divided into / syntactic and semantic paraphrases 30:08
GRI so syntactic paraphrases / involve the same words / or morphologically related words // and they are paraphrased which are broadly applicable 30:19
GRI so they can apply both to / instances of being killed // and instances of hiring and firing // any type of event 30:28
GRI so for example / Booth was &assassina [///] sorry 30:33
GRI Booth assassinated Lincoln 30:35
GRI Lincoln was assassinated by Booth 30:37
GRI the assassination of Lincoln by Booth 30:40
GRI Booth went through with the assassination of Lincoln 30:44
GRI ok? 30:45
GRI with a rather broad notion of &syntac [/] syntactic paraphrase // we can say that these are all syntactic 30:51
GRI we can put a different lexical item here // different word // and we still have a set of / paraphrases 30:57
GRI in contrast / all the paraphrase relations involve different word choices 31:04
GRI assassinated / murdered / fatally shot / in this would be semantic paraphrases 31:11
GRI so / how do we go about attacking these paraphrase relations ? 31:16
GRI the syntactic paraphrases can be addressed / by having deeper syntactic representations // in which we reduce / the paraphrases to a common relationship 31:26
GRI so / we can start with very / simple syntactic relations // for example just finding chunks // finding noun phrases and verb phrases 31:35
GRI then we might go to surface syntax 31:39
GRI we will look at / surface subject / and surface object 31:43
GRI then the next stage / is we might go to / deep structure 31:48
GRI so in deep structure we do logical subject and object 31:51
GRI and / Booth assassinated Lincoln / and Lincoln was assassinated by Booth // could get the same representation 31:59
GRI beyond that we then go to semantic // role structure / even though I'll call this as syntactic paraphrase // also can be described as predicate argument structure 32:11
GRI in a predicate argument structure // we go beyond what's / conventionally called deep structure // and say that the assassination of Lincoln by Booth / even though is a normalization // does have the same basic argument relationship 32:27
GRI the assassination of Lincoln by Booth // is the same argument relationship as the other two instances 32:33
GRI so the deeper we go / the deeper the / level of relationship we capture // the more syntactic paraphrases / upon handle was 32:45
GRI so how do we build analyzers to take care of these / deeper relationships ? 32:51
GRI well nowadays most syntactic analyzers / are created through training from treebanks 32:57
GRI so / with syntactic paraphrases like passives // and actives we'd see lots of examples / even with a limited corpus 33:07
GRI and so / with a treebank we can capture these relationships rather quickly 33:15
GRI the next stage in treebanking / which is now being / actively pursued // is the creation of predicate argument banks 33:24
GRI and there's a lot of work // in the United States and xxx somewhere also in / Europe in building predicate argument banks 33:31
GRI so / in particular there's been work / of English and something called the PropBank // which captures verb relationships // verb argument relationships 33:41
GRI that's been done / at the university of Pennsylvania 33:45
GRI and / we have been working on the NomBank / for noun arguments // so capturing for example relationships between assassination and the assassinated 33:55
GRI all the relationships between / Fred walked for an hour / and Fred took a walk for an hour 34:02
GRI again / nominalization and / verbal form 34:06
GRI so / this predicate-argument banks / will assign common argument labels to a wide range of constructs 34:17
GRI so they'll handle both a verbal // the Bulgarians attacked the Turks 34:24
GRI the Bulgarians' attack on the Turks 34:27
GRI the Bulgarian launched an attack on the Turks 34:29
GRI so all three of these sentences at a predicate-argument level / would become the same structure 34:35
GRI we'll have the same / would be called arg-zero or arg-one in a / predicate-argument bank 34:41
GRI same relationship being / xxx Bulgarians and Turks will appear / for all three sentences 34:47
GRI so / by training an analyzer based on this predicate-argument structure / we can eliminate a good deal of the syntactic paraphrase 34:59
GRI ///but analyze at this predicate-argument level / an then we will look for patterns / in terms of this predicate / argument relationships 35:09
GRI so that's the good news 35:12
GRI the bad news / is we've put another stage into the pipeline // and just go back a second 35:19
GRI remember our said pipeline ? 35:22
GRI well / we've put one more stage on it 35:25
GRI we have before syntactic / we have before surface parsing // and now we've put on one more stage / to do predicate-argument analysis 35:33
GRI so / we get some benefit / but we get some error 35:38
GRI hhh {%act: cough} the deeper the analysis / generally the less accurate the analysis becomes 35:49
GRI we've put in one more stage 35:51
GRI we've &p [/] produced one more stage of errors along with one more stage of analysis 35:57
GRI so / what do we do ? 35:59
GRI this is xxx problem for information extraction deeper 36:03
GRI do we take / a very shallow analysis like chunks / where we might get / ninety five percent accuracy 36:09
GRI or we take / a deep / analysis like predicate-argument structure / which captures much more data // but we might get with the only eighty percent accuracy 36:18
GRI the answer / is both 36:24
GRI so by &mo [/] xxx more complicated system / we can allow patterns at multiple levels 36:32
GRI so we'll write each pattern in terms of the chunk sequence // and the parse tree sequence // and the predicate-argument sequence 36:39
GRI and then we'll use a machine learning method / to weight these different analyses together 36:45
GRI so the hope is / that when the deep analysis fails / when we mess up on the predicate-argument analysis // we'll still be able to make the correct decision from a shallow analysis // finding from the chunks 36:59
GRI so we hope to get coverage / from predicate-argument structure when we get it right // and get accuracy from chunks / when we have / a pattern we've already seen at the chunk level 37:10
GRI so we did some experiments / using this basic approach / in order to / try to discover relation and events 37:19
GRI so / we used what's called a Kernel-based method / in which we measure the similarity between / an example we try to annotate // and one of the examples in the training corpus 37:32
GRI and these kernels work both at the word level // and we have one of the chunk level // and one at the predicate-argument level 37:41
GRI and then we combine / all of these measures / so that anyone of them got a good match // we were able to identify things 37:48
GRI so / the structure looks something like the following 37:52
GRI we have all of these levels of / analysis / logical relations / parsing / name tagging 37:58
GRI we put them into a classifier which does this / generalized matching // and that puts out a result based on what it finds as the best match 38:06
GRI and we got a significant performance / not overwhelming / but a couple of percent gained / in coverage / by being able to match both at the low level / chunk level / syntax level / and predicate-argument level 38:21
GRI and so we think by this approach where we use several levels of analysis // we will get both the benefits / of deep analysis / without a cost / in terms of greater errors 38:32
GRI ok 38:35
GRI so we believe this can / address this / syntactic paraphrase // so we xxx for the semantic paraphrase problem 38:43
GRI so some of the semantic paraphrase can be addressed / by existing lexical resources // such as WordNet 38:59
GRI so in particular people at / Sheffield / measure the degree to which / information extraction patterns could be generalized just using WordNet 39:09
GRI and they measure it on this task / which I've been talking about on and off / for the last few minutes 39:17
GRI this so-called executive succession task // of people being hired and fired 39:21
GRI so you start with a very small seed 39:25
GRI the things people can think of in the first two minutes 39:29
GRI company appointed / elected / promoted / and named a person 39:34
GRI a person resigned / or departed / or quit 39:39
GRI and then / shall explain a moment / we basically use WordNet to generalize from this / co-examples / to see what other / examples of hiring and firing we can find 39:53
GRI and then we won't have some measurement of how effective we are improving coverage 39:57
GRI so we'll use a rather simple metric / so-called text filtering metric // where we see whether / basically what fraction of the sentences which are relevant / are we able to extract 40:11
GRI what fraction of events can we find 40:13
GRI so in Sheffield's experiments / starting with / &ah the seed documents // those just = sorry / the seed patterns 40:26
GRI we would / get hundred percent precision on finding documents 40:31
GRI so every document which matched the pattern was / &ah relevant to the task // but only twenty six percent recall 40:39
GRI by applying WordNet / we then were able to get up to ninety six percent recall / we found almost every document // which was relevant // to hiring or firing 40:50
GRI but unfortunately now only about two thirds / of the documents we found / were relevant 40:55
GRI 'cause we get lots of / &ah / other senses of the words once we start to xxx things 41:03
GRI and if we do the same statistics at the sentence level // we get similar performance so recall goes up from ten to sixty four percent // but the precision finding sentences which are relevant / goes down to forty seven percent 41:16
GRI so / this is fairly effective but / it's not going to do / everything we want by itself 41:24
GRI furthermore the problem is / WordNet is gonna be good for some tasks // and not so good for others 41:31
GRI semantic paraphrase is much more domain-specific than syntactic paraphrase 41:36
GRI so it's hard to prepare some comprehensive resource 41:39
GRI if we suddenly start a talk about genomics // and want to get paraphrase is gonna be much harder to do that using WordNet 41:46
GRI so we turn back to the corpus and say // how can we do things with the corpus ? 41:53
GRI so instead of having people mark the corpus up // can we do things automatically ? 42:00
GRI how can we find / giving a few examples of &peo [/] people being hired // how can we find / automatically from a corpus without marking anything up / other ways of stating the same fact ? 42:14
GRI and we look for a briefly at two approaches here 42:17
GRI predicates with the same arguments // and predicates / in the same documents 42:23
GRI so let's first look at predicates with the same arguments 42:26
GRI ok / the basic intuition / is we find pairs of passages which probably convey the same information 42:36
GRI so we get two newspapers which talk about / hirings and firings on the same day 42:42
GRI and then we align the structures / that points at none correspondence 42:48
GRI so for example Fred XXX Harriet in one newspaper // and Fred YYY Harriet in other newspaper 42:56
GRI and we say / hhh {%act: assent} 42:58
GRI here two patterns which occur between the same names / maybe they are paraphrases / maybe they are relationships 43:07
GRI they / are related terms / because they connect the same people / in news from the same day 43:13
GRI so how accurate this is going to be / depends on part on / &ah / how we pull the texts 43:22
GRI so we have almost parallel text 43:24
GRI I'll explain in a moment 43:26
GRI we were pretty sure they talk about the same things // then we might be able to learn a paraphrase from a single example 43:33
GRI if it's from comparable texts 43:35
GRI so we have some evidence / that this is about the same staff / but we're not sure // then we might use a few examples 43:42
GRI or we can use just / any text / without any constraint / and then we'd need lots of examples 43:48
GRI so in terms of / parallel texts / experiments we'd done in Columbia a couple of years ago / taking two translations of the same novel 43:58
GRI so you would expect if we / take the same novel in French or in Spanish // and translate into English / there'll be a very close correspondence between the two English texts 44:09
GRI so they did this // and they aligned the sentences // and they aligned the individual constituents within the sentences 44:20
GRI and they were able to obtain a number of interesting paraphrases // mostly synonyms // paraphrase of the lexical level / from the translation 44:29
GRI but the problem is / the amount of data you have like this is rather limited 44:35
GRI if you basically have literary data you made of novels which got translated several times 44:40
GRI but whether genomics article is gonna be translated several times from Spanish to English / seems much less likely 44:47
GRI so we have to look at other ways in which we can get / &ah related / texts 44:55
GRI so experiments which we did a couple of years ago at NYU / were based on news stories from multiple sources from the same day 45:02
GRI so we take two newspapers // same day // and we'd looked for / pairs of articles which overlap in terms of several words // particularly several names 45:12
GRI if we find two articles which have a bunch of names in common // then there's a good chance / since they were from the same day / that they talked about the same subject 45:24
GRI ok? 45:24
GRI now we go down into those articles // and we take two -> sentences / which have the same names in them 45:31
GRI if they have the same names in common // then we have a pretty good chance that they're conveying the same information // along maybe with some / related facts 45:42
GRI so we looked then / we keep drilling down into more more detail / we looked for syntactic structures / in the sentences which shared the same names 45:52
GRI and we found sharing two names // we get a paraphrase / precision of sixty two percent 45:58
GRI so this was an experiment about murder in Japanese but / the details perhaps are not so relevant 46:05
GRI we were able to / pull out from news articles without any manual intervention / things which / two thirds of the time were correct / synonyms / correct paraphrases 46:14
GRI so / from single examples we were able to do fairly well 46:24
GRI if we want to increase the accuracy / we can look for multiple examples of the same relationship 46:30
GRI and the basic idea here = ups! / excuse me 46:33
GRI is that xxx an expression appears with several pairs of names 46:37
GRI so we have some / phrase R which appears / between A and B / C and D / E and F 46:43
GRI and then some other expression appears between / also several pairs // A and B / E and F 46:50
GRI then there's a good chance that R and S are paraphrases 46:53
GRI the more examples you find / the better the probability is 46:56
GRI so for example / if we have Eastern Group's agreement to buy Hanson 47:03
GRI Eastern Group / to acquire Hanson 47:06
GRI CBS will acquire Westinghouse 47:09
GRI CBS's purchase of Westinghouse 47:11
GRI CBS agreed to buy Westinghouse 47:12
GRI we pull out the main words // and we look for pairs of main words 47:19
GRI so buy appeared with both CBS / Westinghouse / and Eastern Group / Hanson 47:25
GRI and acquire appeared the same way 47:27
GRI so we say / two examples of each there's a pretty good chance / that we're getting paraphrase 47:33
GRI so there've been a number of / experiments along this line / trying to get the paraphrases 47:40
GRI &ah perhaps the best known fellow is Sergey Brin / who went on to get / billions of dollars doing Google // and isn't worried about / paraphrases anymore // but is hiring / probably dozens of people to worry about paraphrases 47:54
GRI and Lin and Pantel // and some work which / &ah Satoshi Sekine / did at NYU // trying to acquire [/] in each case trying to acquire in this way paraphrases between relations 48:08
GRI and the accuracy we can get if we have enough examples / is quite high 48:14
GRI so we were able to get for example / eighty six percent / accuracy in finding paraphrases / for person-company pairs 48:21
GRI ok 48:26
GRI so this is one approach / to finding / paraphrases with no human intervention 48:31
GRI just / put in millions of texts // and crank away 48:34
GRI the other approach we've been investigating / a frequently / &doc = sorry 48:39
GRI words which occur frequently / co-occur frequently in the same documents 48:44
GRI so here the basic idea / is we start from a topic // get a set of documents on the topic // and get ¶phra [/] get patterns about this topic 48:55
GRI so to explain this / goes back to some work which Ellen Riloff did now ten years ago // on / identifying / paraphrases from relevance judgements for topic // for documents 49:11
GRI so she divided / a large corpus into relevant and irrelevant documents 49:17
GRI at this time / ten years ago / it was about / &ah Latin-American terrorism 49:21
GRI and / classified &m [/] words as people / organizations and so forth 49:28
GRI identified the predicate-argument structures // in the document 49:33
GRI and then count how often / particular structures appear in relevant and irrelevant documents 49:38
GRI and using this / metric shown here / ranked the various constructions 49:45
GRI so the basic intuition is the following 49:48
GRI if we take a bunch of documents // which are about terrorism // and we take another bunch of documents / which are just about everything else / about / cocaine / and politics / and economy // and we ask / which words appear / much more often in the terrorism documents / than appeared in other documents 50:09
GRI and rank words by their relative frequency // then when we'll come to the top of the list // the top of this all divided by illegal / ranking // would be words which are specifically about terrorism 50:21
GRI and most likely we would have / collections of related words 50:26
GRI so I'll show an example in one second / how this works / for the hiring and firing case 50:33
GRI &ah this is an small extension of Riloff's work which was done by / Roman Yangarber at NYU 50:40
GRI basically converting this into bootstrapping method 50:44
GRI to start with a small seed like we did / &ah at Sheffield 50:49
GRI we filled some documents 50:52
GRI picked additional structures with high Riloff metric // and then repeat 50:58
GRI so how do this [/] how does this work ? 51:02
GRI we start with somebody retires // and get all the documents which talk about somebody retiring 51:10
GRI Fred retired / Maki retired / Harry retired 51:14
GRI just collect these retiring documents // and ask / what other / predicates occurred a lot in these documents 51:22
GRI but if you think about articles about retiring // it's very often gonna mention somebody else got hired for the job 51:30
GRI so we look at all the articles // and we look at / what patterns occur / repeatedly / in these retiring articles 51:39
GRI and we see / Harry was named president 51:41
GRI Yuki was named president // and so forth 51:43
GRI so we're just gonna rank the articles 51:45
GRI sorry 51:47
GRI rank the / constructs / by how often they appear in the articles // and pull out the most / commonly occurring constructs 51:55
GRI so we'll pull out / person was named president 52:00
GRI we'd stick it onto the set // and we repeat this process 52:06
GRI each time through the iteration // we'll now retrieve documents which have one of these two patterns // and we'll pick up the third pattern 52:17
GRI we did a very similar experiment what I described for Sheffield // we run another similar seed 52:24
GRI and / we got patterns like this 52:30
GRI again no human intervention 52:32
GRI this is just based upon co-occurrence statistics 52:35
GRI we can find / person succeeded person // person become president // person named president // person joined company // person &le [/] left post 52:47
GRI in terms of performance / statistical performance in terms of finding relevant documents // we got from / eleven percent recall with ninety three percent precision // using just the seed set // to having something like eighty one percent precision / eighty eight percent recall // just doing this / automatic / looping 53:13
GRI so this is comparable to the WordNet-based expansion // and provides a different set of patterns 53:19
GRI so in principle we could combine this / to get better performance 53:23
GRI shown graphically we can see / each step here going from left to right for instance / one iteration // going through / picking up one more / pattern from the set / using the Riloff metric 53:39
GRI and we can see the recall the blue line / rises rather rapidly and then // ups! // and then sort of flattens out 53:50
GRI and the position / slowly declining // going from ninety percent / to something like eighty percent / over the course of these iterations 54:00
GRI so there are a lot of numbers on this graph / but the main thing I wanted to / point out / is we compare the performance / doing this / automatic procedure just described // against the manual procedure // in which / people basically sat there for thirty days / and read newspapers / and try to find all the patterns they could 54:24
GRI and the automatic procedure is quite comparable in performance // to thirty days of manual work 54:31
GRI so they discovered patterns / got about / sixty percent performance 54:38
GRI the manually / collected patterns / I think gonna take some of the audience up next I am not sure 54:45
GRI &ah / the manually collected patterns got fifty six / to sixty four percent performance 54:51
GRI so we can see this automatic procedure can do about as well at finding new patterns / as / people sitting there for / bunch of weeks / collecting data 55:06
GRI so / one word now about / self combining the methods I've just described // the topic based method can find / set of paraphrases quite well / like name appoint select 55:22
GRI but / because they're just looking for articles / rather than patterns which appear in the same articles / they will also find / topically related phrases which are not paraphrases 55:32
GRI so they'll find / group together / appoint and resign / or shoot and die 55:37
GRI so the trick now is to combine the two approaches / we used before 55:44
GRI we talked about / finding patterns because they are in the same articles 55:49
GRI we talked about finding patterns / because they appear with the same arguments 55:53
GRI now we can couple this // this sort of topical discovery / and paraphrase discovery 55:59
GRI first find / all of the topical patterns 56:01
GRI so we'd find retire / and hire / and name and so forth 56:07
GRI and then just put / this set of patterns / into the paraphrase discovery 56:12
GRI and we get a / considerably better result than we could with either method alone 56:16
GRI so we did experiments like this 56:20
GRI we weren't able to find paraphrase / for all of the patterns which were topically relevant 56:26
GRI but we did much better than we could / just finding / using the paraphrase discovery by itself 56:32
GRI so in terms of / &ah numerical performance / we got up to precision by using these two methods together // ninety four percent precision / in terms of / automatically finding patterns which meant the same thing 56:46
GRI it's a rather remarkable level 56:49
GRI &ah / considering / level automation / the coverage is not so good // it's only about forty seven percent of the patterns / which were topically relevant were gathered together 57:01
GRI &ah / it turned out / unfortunately perhaps / we did more experiments and we found out that xxx this works very well / for executive succession // because people normally get hired only one job at a time // we tried it for the / for an arrest domain // where we had / crime stories about people being / in a burglary and theft and being arrested 57:24
GRI and it turns out that / paraphrase doesn't work so well // because people get arrested / typically or / quite often / for several crimes at once 57:35
GRI so Fred was arrested for burglary 57:37
GRI and Fred was arrested for / xxx 57:40
GRI so we combine / all these crimes into one / paraphrase unit 57:44
GRI so there's certainly still a lot of work to do 57:47
GRI whenever a student say / oh! this problem got solved ! 57:49
GRI xxx 57:50
GRI this language problem aren't gonna be solved for another / twenty years / thirty years / up till I retire 57:56
GRI ok 57:59
GRI &ah / Antonio asked me to mention briefly two other topics / before I close here 58:04
GRI so I'll say a few words about this 58:05
GRI one is about cross-language information extraction 58:08
GRI we've done a / number of work 58:11
GRI &ah / at least in the United States / everybody wants to xxx in English // ok? 58:17
GRI nobody wants to read / Spanish / or Chinese / or Arabic / or xxx 58:23
GRI everybody wants English // ok? 58:25
GRI so / we typically get this problem and want to do a database / in language L1 // for language / from texts in L2 58:34
GRI so we get / Spanish / and Chinese / and Arabic texts 58:38
GRI but people say / I want my database in English 58:41
GRI so / how do we do this ? 58:45
GRI well / there're basically two ways of doing this 58:48
GRI we can take the / upper path / or the lower path 58:51
GRI shall explain this two / for a moment 58:54
GRI the upper path / ok / we start with a bunch of articles in / &ah -> Chinese / let's say // ok? 59:05
GRI let's imagine / green is Chinese and red is English 59:08
GRI ok? 59:09
GRI so we start with a bunch of articles in Chinese // and we run / extraction / in Chinese 59:16
GRI we run the Chinese information extraction component 59:19
GRI and we'd get a database in Chinese 59:21
GRI ok? 59:23
GRI then we take / the Chinese database // and we take each terminate // and we run it through some / machine translation system // and get a database in English 59:34
GRI ok? 59:37
GRI other approach / is we take the Chinese texts // and we translate them with a machine translation system / into English 59:48
GRI and then we run an English extraction system 59:51
GRI ok? 59:54
GRI so two ways of addressing the same basic problem 59:59
GRI ok 59:59
GRI which one is gonna work better ? 1:00:01
GRI we take a vote / to vote as a class // like half people raise their hands 1:00:06
GRI but / this is maybe formal lecture / we are not supposed to have people / picking votes here 1:00:11
GRI so study might guess / ok so we did experiments / we had / Japanese graduate students so we did experiments in Japanese // same management succession task 1:00:25
GRI ok? 1:00:25
GRI people being hired or fired 1:00:27
GRI well / we did extraction on / machine translation output 1:00:32
GRI so we took the lower path 1:00:36
GRI we went down with MT / and then across / xxx the red lines 1:00:40
GRI we got forty one percent accuracy 1:00:43
GRI hhh {%act: } 1:00:45
GRI in &cont [/] I ´d come back to the / other items but let's look at the last line 1:00:50
GRI in contrast / if we did / extraction in Japanese // and then just translating the database entrance // we did much better with a sixty percent // which is not much worse than we did / using straight English // there we might get / sixty five or seventy percent performance 1:01:09
GRI so why is it much worse / even you might think 1:01:13
GRI they sell systems to do Japanese translation // they're designed to translate texts // not database 1:01:19
GRI why they should do much worse / if we translate the texts / and then do extraction ? 1:01:24
GRI well / it turns out / the argument structure / is not very well translated by these translation systems 1:01:32
GRI I think for pairs which &ah / have more similar / sentence structure / like Spanish and English // and French and English // you won't have some much / this problem 1:01:43
GRI but / if you / try to do Japanese / or Chinese / or Arabic / or something with a substantially different sentence structure // you'll get the right subject // and you'll get the correct object // and you'll get the correct verb 1:01:54
GRI but they won't be in the correct order // xxx of English 1:01:58
GRI and people sort of compensate / when they read machine translation 1:02:01
GRI and they say / &ah / Fred Smith appointed / IBM president 1:02:08
GRI they all know that wasn't actually that Fred Smith who appointed at / it was &ah / the IBM 1:02:15
GRI but IBM appointed Fred Smith as president 1:02:17
GRI so it's a sort of fix things up 1:02:19
GRI but the extraction system would do that // in so it gets / considering worse performance 1:02:25
GRI so in general / if we can afford the labor to do source language extraction followed by translation // we'll do better 1:02:37
GRI we were able to do some improvement / one of the hard things / in the translation is finding the names 1:02:43
GRI so by doing some projection of structure / from the source language / to the target language // so that we can / identify the names properly in the English texts / by / first finding them in the / Japanese // we were able to do somewhat better 1:03:00
GRI we got a fifty two percent 1:03:02
GRI but still not close to what we can do by doing / genuine foreign language extraction 1:03:08
GRI finally / two slices about / what's going on in terms of multilingual extraction programs in the United States 1:03:17
GRI one is basically an evaluation program 1:03:20
GRI this is a / sneaky trick where they don't / pay people to do research // they just advertise this evaluation 1:03:27
GRI and hope people would do research on this evaluation task // and will all come 1:03:33
GRI so / the United States / we run this so-called ACE evaluation // for doing information extraction 1:03:39
GRI and / until now / until this / past year it was in Arabic / in Chinese / in English 1:03:47
GRI ok? 1:03:48
GRI you can sort of imagine why the United States is interested in Arabic and Chinese and / English 1:03:53
GRI &ah / and we may / wonder / but / &ah Spanish has been added as a fourth language for the evaluation this year 1:04:02
GRI so if anybody has a good information extraction system for Spanish // they are welcomed to come to Washington 1:04:09
GRI &ah this evaluation is being run the first day of February of two thousand seven // and participate 1:04:17
GRI completely open / for the / Chinese task / we have a number of Chinese universities coming // participate in the evaluation 1:04:25
GRI &ah the Spanish task for this year would basically be / finding entities 1:04:29
GRI which means if you have a good / chunker / and you have good coreference in Spanish // you can probably do good job at this task 1:04:37
GRI other multilingual IE program / &ah / is a research program / a large scale research program founded by DARPA // and begun just one year ago // called GALE // also trilingual // Arabic / and Chinese / and English 1:04:54
GRI and involves / all the NLP processing you &cou [///] one 1:04:59
GRI so it involves / automatic speech recognition / machine translation / information retrieval / information extraction / a little bit of summarization / almost everything 1:05:09
GRI so the / retrieval extraction task involves answering questions about a multilingual corpus 1:05:18
GRI so we give them / for example / &ah / Arabic Al Jazeera / &ah / TV broadcasts 1:05:26
GRI and we're supposed to do / automatic / ASR / followed by machine translation / followed by extraction 1:05:34
GRI and xxx to answer questions like / describe reaction of country to event 1:05:39
GRI or list attacks in location during time and related deaths 1:05:43
GRI so a sort of mix of summarization is supposed to do extraction to pull out the relevant sentences // and then build the summary around them 1:05:52
GRI and this is / as I was saying the design / United States' every sponsors only in English 1:05:59
GRI people don't want to talk anything but English in the United States 1:06:02
GRI so we are supposed to / pull out / this / Al Jazeera staff 1:06:07
GRI transcribe it // translate it // present English sentences which are relevant to answering / these particular questions 1:06:15
GRI and DARPA says by the time / five years have finished we are supposed to do better than people do at these tasks 1:06:21
GRI so / I don't know if we believe that / but / they try to scare us every year with / very high performance targets 1:06:29
GRI so we are looking / in order to reach this level of performance / to combine the different IE paths 1:06:36
GRI we think that will have to go both ways / and combine the results at the end // in order to get the best extraction performance 1:06:43
GRI ok 1:06:47
GRI so to conclude / the current IE technology which I've / been describing / provides a powerful tool / for targeted searches in large text collections 1:06:57
GRI so I think it provides a / real capability beyond what we ever want to get / from information retrieval 1:07:03
GRI but it has limitations of performance // and recent basic research on NLP methods / some of which I've described here / offer substantial opportunities for improving extraction performance / and portability 1:07:17
GRI and the main points I've talked about a global optimization for improving / performance levels / treebanks / for things like predicate-argument level // &pre [/] predicate-argument / discovery // and corpus-based discovery methods / for greater coverage 1:07:33
GRI thank you 1:07:34
