When try use CsvReader<CsvRecord> records second time in separate method then records is empty #106

Activity

1. dbulahov

   changed the title [-]When try use CsvReader<CsvRecord> records in private method then records is empty[/-] [+]When try use CsvReader<CsvRecord> records second time in separate method then records is empty[/+] on Jan 17, 2024
2. osiegmar commented on Jan 17, 2024

   

   osiegmar

   on Jan 17, 2024

   Owner

   More actions

   Your example demonstrates the following:

   try (CsvReader<CsvRecord> csv = CsvReader.builder().ofCsvRecord(file)) {
       // first loop
       for (CsvRecord r : csv) {
           System.out.println(r);
       }
   
       // second loop
       for (CsvRecord r : csv) {
           System.out.println(r);
       }
   }

   The used for-each loop is a short version of the following:

   try (CsvReader<CsvRecord> csv = CsvReader.builder().ofCsvRecord(file)) {
       Iterator<CsvRecord> csv = csv.iterator();
   
       // first loop
       while (iterator.hasNext()) {
           System.out.println(iterator.next());
       }
   
       // second loop
       while (iterator.hasNext()) {
           System.out.println(iterator.next());
       }
   }

   The first loop displays all records, but the second loop is empty. This is expected, and FastCSV evidently requires additional documentation in this regard.

   The call to CsvReader.builder().ofCsvRecord(file) returns an Iterable of CsvRecord (CsvReader). With each iteration of the loop, you read (consume) one record from the file. After all records are read, there are no more records.

   If you truly want to store all records of the file for repeated access, you'd have to collect them:

   try (CsvReader<CsvRecord> csv = CsvReader.builder().ofCsvRecord(file)) {
       final List<CsvRecord> records = csv.stream().toList();
   
       // first loop
       for (CsvRecord r : records) {
           System.out.println(r);
       }
   
       // second loop
       for (CsvRecord r : records) {
           System.out.println(r);
       }
   }

   Keep in mind that with this approach, all records of the file have to be kept in memory, while the streaming approach of FastCSV allows you to read huge CSV files with only a little memory consumption.
3. osiegmar

   added a commit that references this issue on Jan 17, 2024

   Clarify/Document iterative read process #106

   cb616ce
4. osiegmar

   closed this as completedon Jan 17, 2024
5. osiegmar

   added

   question

   on Jan 17, 2024
6. stbischof commented on Jan 17, 2024

   

   stbischof

   on Jan 17, 2024

   More actions

   When i remember currently the Usercase is to handle csv with typed columns.

   col1,col2
   String,int
   Foo,1
   Bar,987
   ...

   there you need information from header and line 2.

   Then run fast over all the following lines.
   What woulr be fastest way to handle this?
7. osiegmar commented on Jan 18, 2024

   

   osiegmar

   on Jan 18, 2024

   Owner

   More actions

   The fastest way would be a custom callback handler – see this example. But I wouldn't recommend that way unless you absolutely need the last bit of performance.

   A very fast and maintainable way would be something like this:

   private static void readFile(Path file) throws IOException {
       // Make sure we have the same number of fields in each record
       CsvReader.CsvReaderBuilder builder = CsvReader.builder()
           .ignoreDifferentFieldCount(false);
   
       try (CloseableIterator<CsvRecord> it = builder.ofCsvRecord(file).iterator()) {
           if (!it.hasNext()) {
               throw new IllegalStateException("No header found");
           }
           List<String> header = it.next().getFields();
   
           if (!it.hasNext()) {
               throw new IllegalStateException("No data types found");
           }
           List<String> dataTypes = it.next().getFields();
   
           while (it.hasNext()) {
               YourCustomRecord yourCustomRecord = mapRecord(header, dataTypes, it.next());
   
               // do something with your record
               System.out.println(yourCustomRecord);
           }
       }
   }
   
   private static YourCustomRecord mapRecord(List<String> header, List<String> dataTypes, CsvRecord record) {
       // Map the record to your needs with the given header and data types
       return new YourCustomRecord(record.getField(0), Integer.parseInt(record.getField(1)));
   }
   
   private record YourCustomRecord(String s, int i) {
   }

   You may also use NamedCsvRecords (.ofNamedCsvRecord(file)) if you want to access the fields by name.
   The efficiency of your mapRecord implementation will crucial to the performance of the whole process. But that highly depends on your exact needs.

   In the future, with Stream Gatherers (JEP 461, a Preview Feature of Java 22) we'll also be able to implement this using Java streams. The following codes uses build 22-ea+31-2314 of OpenJDK:

   private static void readFile(Path file) throws IOException {
       // Make sure we have the same number of fields in each record
       CsvReader.CsvReaderBuilder builder = CsvReader.builder()
           .ignoreDifferentFieldCount(false);
   
       try (Stream<CsvRecord> stream = builder.ofCsvRecord(file).stream()) {
           stream
               .gather(Gatherer.ofSequential(new YourCustomIntegrator()))
               .forEach(yourCustomRecord -> {
                   // do something with your record
                   System.out.println(yourCustomRecord);
               });
       }
   }
   
   private static class YourCustomIntegrator implements Gatherer.Integrator<Void, CsvRecord, YourCustomRecord> {
   
       private List<String> header;
       private List<String> dataTypes;
   
       @Override
       public boolean integrate(Void state, CsvRecord element,
                                Gatherer.Downstream<? super YourCustomRecord> downstream) {
           if (dataTypes != null) {
               return downstream.push(mapRecord(header, dataTypes, element));
           }
   
           if (header == null) {
               header = element.getFields();
           } else {
               dataTypes = element.getFields();
           }
           return true;
       }
   
   }
8. dbulahov commented on Jan 18, 2024

   

   dbulahov

   on Jan 18, 2024

   Author

   More actions

   Thank you very much for your help
9. Repository owner locked and limited conversation to collaborators on Jan 18, 2024
10. osiegmar

    converted this issue into a discussion #107 on Jan 18, 2024